Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
Version 2 submitted March 04, 2021
ACTIVITY MONITORING,
EVALUATION, AND LEARNING PLAN
ROUTINE IMMUNIZATION TRANSFORMATION AND EQUITY
M-RITE Activity Monitoring, Evaluation, and Learning Plan 2
TABLE OF CONTENTS Abbreviations 3
Introduction 4
Alignment with MOMENTUM Knowledge Accelerator 5
Objectives 5
Results Framework 6
Performance Monitoring 8
Indicators 8
Data Management and Reporting 16
Data Management and Quality Assurance 16
Reporting 16
Data Analysis and Use 17
Collaborating, Learning, and Adapting 18
Learning Agenda and Evaluation 19
MEL Roles and Expectations 20
Appendix A: Contextual Indicators 21
Appendix B: Prioritized Performance Indicator Reporting Sheets 26
Appendix C: Learning Agenda, Themes, and Questions 40
Appendix D: Supplementary Indicators 44
References 59
M-RITE Activity Monitoring, Evaluation, and Learning Plan 3
ABBREVIATIONS
AMELP Activity Monitoring, Evaluation, and Learning Plan
CLA collaborating, learning, and adapting
CRaD Country Readiness and Delivery
DHIS-2 District Health Information Software-2
EIR electronic immunization record
EPI expanded program on immunization
FP family planning
GAVI Gavi, the Vaccine Alliance
HCD human-centered design
HMIS health management information system
IA2030 Immunization Agenda 2030
IR intermediate result
MOMENTUM Moving Integrated, Quality Maternal, Newborn, and Child Health and Family Planning and
Reproductive Services to Scale
M-RITE MOMENTUM Routine Immunization Transformation and Equity
M&E monitoring and evaluation
MCSP Maternal and Child Survival Program
MEL monitoring, evaluation, and learning
MKA MOMENTUM Knowledge Accelerator
MNCH maternal, newborn, and child health
PHC primary health care
RH reproductive health
RI routine immunization
TOC theory of change
UNICEF United Nations Children’s Fund
USAID United States Agency for International Development
WHO World Health Organization
XM cross-MOMENTUM
M-RITE Activity Monitoring, Evaluation, and Learning Plan 4
INTRODUCTION
MOMENTUM Routine Immunization Transformation and Equity (M-RITE) is one of a suite of the United States
Agency for International Development (USAID) projects designed to increase the capacity of host-country
institutions and local organizations to introduce, deliver, scale up, and sustain the use of evidence-based,
high-quality maternal, newborn, and child health (MNCH) services; voluntary family planning (FP); and
reproductive health (RH) care. M-RITE will apply best practices and explore innovations to increase equitable
immunization coverage in USAID-supported countries around the globe, with a particular focus on reducing
the number of zero-dose children. M-RITE will also contribute to mitigating the effects of COVID-19 on
immunization services and support the eventual introduction of a COVID-19 vaccine. Working at multiple
levels and in alignment with global strategies such as the Immunization Agenda 2030 (IA2030) and Gavi 5.0,
M-RITE will focus on sustainably strengthening routine immunization (RI) programs by identifying and
addressing persistent bottlenecks and barriers in the planning, delivery, demand, and uptake of
immunization services. The theory of change (TOC) in Figure 1 describes M-RITE’s desired outcomes and their
relationships to the drivers of equitable immunization coverage. The TOC identifies the strategies that M-RITE
will deploy and the contextual challenges (including COVID-19) that influence health outcomes.
FIGURE 1: M-RITE THEORY OF CHANGE
This Activity Monitoring, Evaluation, and Learning Plan (AMELP) describes the tools that M-RITE will use to
support its approaches to monitoring, evaluation, and learning (MEL), and our overall methodology for
generating evidence, supporting adaptive management and learning, and achieving results. M-RITE’s AMELP
reflects the guiding principles of measurement for performance monitoring and accountability, measurement
for learning and program improvement, and measurement for advocacy.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 5
The M-RITE AMELP aligns with USAID’s Collaborating, Learning, and Adapting (CLA) practices to inform
project strategy and implementation to accelerate sustainable progress toward the intended result of
improved immunization coverage and equity. The AMELP aligns with USAID’S ADS 201, the Evaluation Policy,
and the CLA Toolkit.1,2,3 This AMELP was also developed in consultation with MEL counterparts from IA2030
and Gavi, and aligns where possible to those key immunization monitoring and evaluation (M&E) frameworks
to support principles of donor harmonization, alignment with local priorities and processes, and local
ownership in support of the Journey to Self-Reliance, as well as process indicators tested during the Maternal
and Child Survival Program (MCSP). We draw indicators and data management tools from country health
management information systems (HMIS) and other established reporting and data management
mechanisms to support country ownership and use of project-supported data.
ALIGNMENT WITH MOMENTUM KNOWLEDGE ACCELERATOR
In accordance with the letter of commitment between M-RITE and MOMENTUM Knowledge Accelerator
(MKA), the two MOMENTUM projects will work closely together. For example, the AMELP aligns with MKA’s
existing approaches, tools, and indicators, including the cross-MOMENTUM (XM) M&E framework and
learning questions, and guidance on data sharing and data systems. M-RITE’s AMELP benefitted from
available MKA tools including its technical brief on complexity-aware monitoring and adaptive learning toolkit
draft. M-RITE will work with MKA on analysis and dissemination, and on data collection as needed. These
linkages are noted throughout this report.
OBJECTIVES
The main objectives of the M-RITE AMELP are to:
● Support the M-RITE team to monitor and evaluate the project’s implementation and progress
toward its intended and intermediate results (IRs).
● Enable ongoing adaptive learning during activity design and implementation—including the
development and implementation of learning questions—to identify what is and is not working,
make mid-course corrections, and improve the overall impact of M-RITE.
M-RITE’s MEL approach aims to achieve maximum learning that can be applied immediately to the project
goal of improving immunization equity and coverage. Recent stagnation in immunization coverage and
persistent under-immunization among certain populations requires a MEL approach that focuses on the root
causes of under-immunization, for example through new indicators supplemented by qualitative information;
the use of equity-driven analyses to stimulate action; and learning questions that target the most entrenched
obstacles. M-RITE will contribute to improving immunization equity and coverage in USAID-priority countries
and any others that buy into the project. We will collaborate with other stakeholders to strengthen health
system capacity and performance, tracking activities and outputs and measuring changes in outcomes, where
feasible. In the case of national-level outcome and impact-level results, however, it will not be possible to
attribute specific intervention coverage increases and mortality reductions to the project.
The AMELP includes M-RITE’s results framework, performance monitoring indicators, learning agenda, and
complementary M&E strategies. The AMELP also includes the project’s data quality assurance, data
management and reporting systems, and M-RITE’s approach to CLA practices.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 6
M-RITE will update indicators, tools, and our underlying TOC as we learn. The AMELP is a living document and
will be updated yearly as needed. Throughout the project, the senior management team will review
indicators with USAID to ensure they conform to current USAID guidelines, reflect changing context, and
remain relevant for adaptive management.
RESULTS FRAMEWORK All MOMENTUM awards have four identical results and IRs:
● Result 1: Access to and use of evidence-based, quality MNCH/FP/RH information, services, and
interventions scaled-up and sustained.
o IR 1.1: Improved service readiness to provide quality MNCH/FP/RH interventions in public
and private sectors, including emergency care.
o IR 1.2: Improved MNCH/FP/RH practices at the individual, family, and community level.
o IR 1.3: Increased demand for and utilization of quality MNCH/FP/RH interventions and care
by individuals, families, and communities.
● Result 2: Capacity of host-country institutions, local organizations, and providers to deliver evidence-
based, quality MNCH/FP/RH services improved.
o IR 2.1: Increased effectiveness of country institutions and systems to sustainably plan and
manage quality MNCH/FP/RH services and care, including anticipating and responding to
crises, as well as long-term demographic and epidemiologic transitions.
o IR 2.2: Enhanced capacity of country governing bodies to align MNCH/FP/RH policies and
guidelines with international standards and evidence-based, innovative, and promising
practices.
o IR 2.3: Strengthened engagement with civil society, local institutions, and community-based
and faith-based organizations to narrow MNCH/FP/RH equity gaps and quality of care and
hold health system accountable.
● Result 3: Adaptive learning and use of evidence in MNCH/FP/RH programming through sustained
host-country technical leadership increased.
o IR 3.1: Increased appropriate and timely availability and use of data for decision-making in
MNCH/FP/RH policy and programs at the global, regional, and sub-national country levels.
o IR3.2 Increased knowledge generation, translation, and management strategies adopted to
support best practices in MNCH/FP/RH policies and programs expanded at the global,
regional, national, and sub-national country levels.
o IR 3.3: Testing and adoption of innovative practices, including the use of digital health
technologies, to improve MNCH/FP/RH outcomes increased.
o IR3.4 Contributions to USAID’s global technical leadership in MNCH/FP/RH by USAID-funded
partners and USAID-supported countries increased.
● Result 4: Cross-sectoral collaboration and innovative partnerships between MNCH/FP/RH and non-
MNCH/FP/RH local organizations increased.
o IR4.1: International and national public-private partnerships increased.
o IR 4.2: Health partnerships with educational institutions expanded.
o IR 4.3: Health partnerships with corporate and philanthropic organizations increased.
o IR 4.4: Health and non-health organization partnerships expanded.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 7
M-RITE will contribute to the ultimate health impact goal through reduced rates of zero-dose children,
reduced rates of under-vaccination, and strengthened country capacity to mitigate entrenched obstacles and
advance the Journey to Self-Reliance.
The Results Framework in Figure 2 illustrates how implementation of M-RITE’s planned activities are
expected to contribute to the achievement of MOMENTUM IRs and results. In line with the XM MEL
framework, Results 2–4 contribute to achieving Result 1. Activities are designed to contribute to multiple IRs,
recognizing the interdependence within and across activities, IRs, and results. Figure 2 uses multiple arrows
between boxes to demonstrate that multiple activities or IRs are often necessary to achieve a given result.
Feedback loops, particularly from Result 3, demonstrate the need for adaptive learning to optimize
outcomes.
Figure 2 also reflects M-RITE’s holistic and cross-cutting use of project strategies. Guided by systems thinking,
immunization expertise, and a human-centered design (HCD) approach to identify and understand the
perspectives of families, communities, and health workers and the drivers of their immunization-related
behavior, M-RITE uses the following strategies to design and deliver project activities:
● Catalyze country leadership and commitment to equity.
● Foster resilient systems and communities.
● Partner for impact.
● Use data better.
● Maintain, adapt, reinstate, and improve RI services.
FIGURE 2: RESULTS FRAMEWORK
M-RITE Activity Monitoring, Evaluation, and Learning Plan 8
PERFORMANCE MONITORING
M-RITE’s performance monitoring approach identifies indicators and associated data sources along the
results framework to monitor multiple steps of a given activity’s anticipated causal chain, identify potential
gaps and drivers of success, and inform implementation adaptations as early as possible.4 M-RITE’s goal is for
performance monitoring to result in adaptive learning. Occasionally, M-RITE’s approach to performance
monitoring and choice of indicators will serve an advocacy purpose, such as through the inclusion of birth
dose vaccination indicators that M-RITE may not have full responsibility to affect. M-RITE’s learning-oriented
performance monitoring approach will:
● Draw on national- and district-level HMIS/District Health Information Software 2 (DHIS-2) data to
measure its own performance, strengthen existing systems, and avoid duplication.
● Surface relevant, timely, and useful information that can directly influence learning and project
improvement.
● Enable ownership and engagement of project team members and other stakeholders who are
responsible for, can learn from, and can take action to improve activity design, implementation, and
outcomes.
● Ensure team understanding of process and context so that why something works, or does not, is as
apparent as whether it works.
M-RITE’s performance monitoring approach is informed by best practices from causal link monitoring to
maximize adaptive learning and management.4 We will use principles and tools from causal link monitoring
to identify inputs, processes, and assumptions necessary to achieve desired outputs and outcomes; improve
activity design based on this reflection; and monitor and adapt as real-world chains unfold. We will focus
causal link monitoring efforts on activities in which the causal chains have greater uncertainty (e.g., because
the intervention or system is complex), and will build detailed logic models with staff and stakeholders to
identify assumptions necessary for the achievement of the desired outcomes. This will help identify process
and contextual indicators that are most critical to monitor during activity implementation, which in turn will
enable faster adaptive management of changes and unintended consequences. Where program design
allows, M-RITE will use quasi-experimental and qualitative methods such as interrupted time-series,
difference-in-difference, and contribution analysis to estimate M-RITE’s contribution to observed changes in
outcomes. Rich qualitative data and learning-focused case studies will help to explain observed indicator
trends or estimates of M-RITE’s contributions.
In the dynamic context of COVID-19, the use of complementary and complexity-aware monitoring will be
critical to identify emerging causal pathways and change mechanisms within this project and immunization
programs (e.g., to monitor and understand the unintended consequences of revised service delivery
guidelines or new immunization service delivery models). Appendix A includes indicators and data sources to
monitor context including social, economic, and political drivers, and indicators to monitor COVID-19’s
consequences on demand for and care-seeking of immunization services, vaccine and health worker
availability, patient access to clinics, and service quality.
INDICATORS
The AMELP includes performance monitoring indicators that will be used to measure outputs, initial outcomes, and outcomes that contribute to or reflect IRs and results at baseline, and then to monitor their change in response to M-RITE activities. In addition to these traditional performance indicators, we also include indicators that are included for advocacy purposes (marked with an “A” in the table) and indicators
M-RITE Activity Monitoring, Evaluation, and Learning Plan 9
that will be used primarily for learning purposes (marked with an “L” in the table). Indicators that require tailoring to country needs and context to be most useful are marked with a “T”. The indicators are organized according to results and reflect anticipated life-of-project activities and outcomes. The indicator table below contains information on indicator type, the IR it measures, data source, and USAID reporting frequency. The Performance Indicator Reference Sheets (PIRS) in Appendix C provide full definitions, rationale for indicators, data sources, and disaggregation (addressing the project’s core principles). Many indicators were drawn from partners such as Gavi, IA2030, MCSP, the UNICEF Equity Reference Group, as well as relevant indicators from the XM MEL framework. M-RITE will also monitor supplemental performance indicators (Appendix D) and activity-specific monitoring indicators. These indicators, while not reported to USAID, will be used to monitor the project and help to tell the M-RITE story. The AMELP recognizes indicators may be modified according to countries’ indicator definitions and data
availability. M-RITE will use routine data (e.g., HMIS, DHIS-2, electronic immunization records [EIRs]) where
possible to support near real-time AMEL, but will supplement it with estimates and survey data, particularly
in countries in which data quality is a significant concern.
TABLE 1: M-RITE PERFORMANCE MONITORING INDICATORS
Code Indicator name (* indicates dependent on country scope)
Type Definition notes (see PIRS for full definitions)
Data source Indicator source
Disaggregation Reporting frequency
Result 1: Access to and use of evidence-based, high-quality MNCH/FP/RH information, services, and interventions scaled-up and sustained
1.0.1 Dropout between first dose (DTP1) and third dose (DTP3) of DTP-containing vaccines in M-RITE- supported areas.±
Outcome The difference between the number or percent of children <12 months who received DTP1, but not DTP3 vaccine.
HMIS XM By SDP, country, urban/rural, district
Annual
1.0.2 Number or percent of children <12 months who received DPT3/Penta 3 vaccine in M-RITE -supported areas.
Outcome HMIS XM By SDP, country, urban/rural, district
Annual
1.0.3 Number or percent of surviving infants who received 1st dose of measles-containing vaccine (MCV) in M-RITE -supported areas.
Outcome
HMIS XM By SDP, country, urban/rural, district
Annual
1.0.4
Number or percent of surviving infants who received 2nd dose of measles-containing vaccine (MCV2) in M-RITE -supported areas.
Outcome HMIS XM By SDP, country, urban/rural, district
Annual
1.0.5 Number or percent of surviving infants who did not receive 1st dose of DPT/Penta vaccine in M-RITE-supported areas.
Outcome Zero-dose indicator
HMIS IA2030 By SDP, country, urban/rural, district
Annual
1.0.7 Number or percent of live births who received a birth
dose of BCG (BCG0) in M-RITE-supported areas. (A)
Outcome HMIS IA2030 By SDP, country, urban/rural, district
Annual
1.0.8 Equity: DTP3 coverage among historically unreached
communities or group. (T)
Outcome This indicator will be tailored to national and sub-national context and “communities” or “groups” will depend on a specific national or sub-national drivers of under-immunization or zero-dose.
HMIS or survey
Depends on country tailoring
Annual
IR 1.1: Improved service readiness to provide quality MNCH/FP/RH interventions in public and private sectors, including emergency care
M-RITE Activity Monitoring, Evaluation, and Learning Plan 11
1.1.1 Percent of M-RITE-supported districts (or facilities) reporting full stock availability at service delivery level during re-supply cycle or reporting period.
Initial outcome
Full stock availability is defined as the absence of any stock-out for vaccines and supplies.
LMIS, HMIS, or other stock databases
IA2030 LMIS, HMIS, or other stock databases
Annual
1.1.3 Percent of health facilities with evidence of utilization of an up-to-date and quality microplan to identify at-risk or vulnerable communities and children.*
Initial outcome
"Up-to-date" will be defined according to the national recommendations and context. “At-risk” will be defined according to national and sub-national context and M-RITE’s assessment.
Project records
MCSP process indicators
Country, district, SDP type
Annual
1.1.6 Percent of health workers in M-RITE-supported facilities who can explain how to identify and reach zero-dose and under-immunized children.*
Initial outcome
Project records
Country, district, SDP type, gender of health worker
Annual
1.1.7 Percent of M-RITE-supported health facilities who improve their readiness to reach zero-dose and under-immunized children with high-quality immunization services. (T)
Outcome “Readiness” will be defined in country PMPs based on needs identified through M-RITE’s baseline assessment.
Project records
Country, district, SDP type, public/private
Annual
IR 1.3: Increased demand for and utilization of quality MNCH/FP/RH interventions and care by individuals, families, and communities
1.3.1 Percent of M-RITE-supported areas with activities led by the EPI program designed and implemented to overcome specific vaccine demand- or utilization-related barriers to immunization in high-risk communities.*
Initial outcome
"High risk" will be defined at the level of planning and implementation, ideally district or facility through M-RITE’s co-creation process.
District-level EPI program workplan reviews, rapid telephone surveys
IA2030 Country, district, type of SDP (as relevant)
Annual
Result 2: Capacity of host-country institutions, local organizations, and providers to deliver evidence-based, quality MNCH/FP/RH services improved, institutionalized, measured, documented, and responsive to population needs
2.0.2 Percent of national and sub-national immunization stakeholders whose capacity* was strengthened** by M-RITE-supported capacity strengthening activities. (T) *Capacity will be defined and tailored to specific country contexts
Output Capacity building interventions are defined as any forms of interventions or activities that improve the ability of a person, group, or organization to meet objectives or to perform better. This indicator counts those stakeholders who were reached by such interventions.
Project records
Country, stakeholder type
Annual
M-RITE Activity Monitoring, Evaluation, and Learning Plan 12
IR 2.1: Increased effectiveness of country institutions and systems to sustainably plan and manage quality services/care, including anticipating and responding to crises, as well as long-term demographic and epidemiologic transitions
2.1.3 Evidence of increased use of sub-national (e.g. district-level) processes and tools* to regularly review and address immunization performance and equity, including gender equity, and to anticipate and respond to change. (T)
Initial outcome
*Regular review processes and tools will be defined and tailored to country context but may include: quarterly review meetings, data review meetings, effective planning processes, uptake of new global guidance, microplanning, use of costing or budgeting tools.
Project records
Country Annual
IR 2.2: Enhanced capacity of country governing bodies to align policies and guidelines with international standards and evidence-based, innovative, and promising practices
2.2.2 Number of global, national, and sub-national policies (including strategies, guidelines, operational plans)
developed with M-RITE support related to immunization
services or MOMENTUM cross-cutting areas.
Initial outcome
Project records
MCGL Country, type of policy, topic (e.g., COVID-19 vs. non-COVID-19)
Annual
2.3.1 Percent of M-RITE-assisted local organizations with improved performance.*
Outcome Capacity is the ability of people,
organizations, and society as a whole to
manage their affairs. Capacity
development is the process of
strengthening and maintaining such
capacity. Capacity is a form of
potential; it is not visible until it is used.
Therefore, performance is the key
consideration in determining whether
capacity has changed. Organizations
with improved performance will have
undergone a deliberate process to
improve execution of organizational
mandates to deliver results its
stakeholders.
M-RITE will measure improvement
organization using already existing
tools such as university accreditation
Project records
USG CBLD-9
Country, Type of organization
Annual
M-RITE Activity Monitoring, Evaluation, and Learning Plan 13
process, ISO professional farmer
organization guidelines, OCA tool, etc.).
And for monitoring and measuring
changes on a key performance metric
(e.g., increased sales, reduced staff
turnover, increased efficiency of
service delivery, etc.).
Result 3: Adaptive learning and use of evidence in MNCH/FP/RH programming through sustained host-country technical leadership increased
3.0.1 Number of evidence products, tools, or data analyses produced by M-RITE that include actionable recommendations and an equity and gender equity lens.
Output "Actionable recommendations" will be measured through existing tools.
Project records
MKA Topic (e.g., COVID-19 or non-COVID-19), country
Semi-annual
3.0.4 Evidence of M-RITE adaptive learning or HCD approaches contributing to improved design or implementation of global, national, or sub-national immunization policies, programs, or interventions.
Outcome Case studies n/a Annual
3.0.5 Evidence of increased use of equity-related data and evidence by M-RITE-supported EPI programs.
Outcome Case studies n/a Annual
IR 3.1: Increased appropriate and timely availability and use of data for decision making in MNCH/FP/RH policy and programs at the global, regional, and sub-national country levels
3.1.7 Evidence of increased use of routine immunization data for EPI-program and policy decision-making.
Outcome Case studies n/a Annual
IR 3.2: Increased knowledge-generation, translation, and management strategies adopted to support best practices in MNCH/FP/RH policies and programs expanded at the global, regional, national, and sub-national country levels
M-RITE Activity Monitoring, Evaluation, and Learning Plan 14
3.2.2 Number of individuals reached through M-RITE co-creation, dissemination, knowledge exchange, and knowledge translation activities and events.
Initial outcome
Individuals are counted in “reach" if they were the active participant in an M-RITE co-creation or knowledge-related event. Active participation could be defined as those people who actively have to choose to engage in content - by attending meetings, opening e-mails, logging into webinars, etc.
Project records
Similar to MKA
Organization, gender (where possible), country
Semi-annual
IR 3.3: Testing and adoption of innovative practices, including the use of digital health technologies, to improve MNCH/FP/RH outcomes increased.
3.3.1 Number of innovations supported through M-RITE assistance.*
Output "Supported" refers to technical, in-kind, or financial support from M-RITE related to the introduction, testing, or adoption of innovations.
Project records
USG STIR10
Country, topic, new or ongoing support, innovation phase
Annual
IR 3.4: Contributions to USAID’s global technical leadership in MNCH/FP/RH by USAID-funded partners and USAID-supported countries increased
3.4.2 Number of global and regional documents (guidelines, strategies, tools, etc.) that M-RITE contributes to.
Initial outcome
Project records
Topic (e.g., COVID-19 or non-COVID-19), new or ongoing
Semi-annual
Result 4: Cross-sectoral collaboration and innovative partnerships between MNCH/FP/RH and non-MNCH/FP/RH organizations increased
4.0.2 Number of multi-sectoral collaborations formed or strengthened at global, regional, national, and sub-national levels as a result of M-RITE support to address immunization coverage and equity issues.*
Initial outcome
A "collaboration" is a relationship between two organizations working together with the goal of jointly producing an output or contributing to a common goal.
Project records, case studies
Collaboration type (aligned with IRs), topic, country, new (formed) or existing (strengthened), level
Annual
4.0.3 Number of partnerships formed or strengthened at global, regional, national, and sub-national levels to address immunization coverage and equity issues.*
Initial outcome
Project records, case studies
Collaboration type (aligned with IRs), topic, country, new (formed) or existing (strengthened), level
Annual
Table notes: ± "M-RITE-supported areas" will be defined based on the scope of M-RITE's engagement in a country and that country's specific administrative units.
*Indicators dependent on scope of country awards.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 16
DATA MANAGEMENT AND REPORTING
DATA MANAGEMENT AND QUALITY ASSURANCE
M-RITE will implement a high-quality data management system that offers consistent data collection,
systematic data cleaning, and secure data storage. The project will develop and maintain a project database
on a DHIS-2 instance that is interoperable with the MKA XM data system. The system will follow the
standards of the data exchange process between MKA and M-RITE to enable data exchange and integration.
These data will be entered manually but the exchange will be automatic, as guided by the document. The M-
RITE database is also being designed based on lessons from the development of the MKA data system. The
project database will include baseline values for some performance indicators, target values for each
indicator once the final AMELP is approved, and definitions of the indicators. For service delivery-related
indicators, including those from HMIS or logistic management information systems (LMIS), the unit of
observation in most cases will be a health district (or similar administrative unit), the lowest level of data
aggregation that is most readily available in most countries’ HMIS. Activity leads will input aggregate data
from various sources into the database quarterly.
To ensure that information gathered is credible and usable, it must be extracted from a high-quality reporting
system. Data quality measures necessitate that data are accurate, reliable, timely, relevant, and complete.
Such parameters require placement of data quality measures throughout the data flow and management
system. M-RITE will design a system that ensures internal data quality checks. We will design standard
operating procedures for data collection, entry, and storage. In addition, a list of internal data quality checks
will be built into the data storage system to ensure consistency across each indicator. We will pay special
attention to additional data quality issues because of COVID-19 interruptions. M-RITE’s senior management
team will review data for completeness on a quarterly basis, and conduct spot checks to ensure data quality.
Data validation dashboards will be used to catch reporting-level errors.
M-RITE will continually consider known data quality issues in immunization. M-RITE and XM indicators
related to vaccine coverage suffer common data quality issues in many countries. One is that population
denominators (i.e., the estimated target population) are often inaccurate, leading to inaccurate coverage
estimates. To avoid this, M-RITE will report both the number of vaccines administered (numerator) and
coverage (numerator/denominator). Where applicable, we will apply denominator correction formulas
developed during MCSP.5 To avoid misinterpreting late or under-reporting into DHIS-2 during COVID-19, M-
RITE will track data reporting indicators to develop country- or district-level approaches to reporting and
interpreting data. M-RITE will also support data quality assessments and improvement plans where feasible
and where data quality and use is a significant bottleneck not only to M-RITE performance monitoring, but to
equitable immunization coverage more generally.
REPORTING
The M-RITE data system will be used to compile data on the indicators above at the frequency drafted in the
indicator table. Indicator reporting to the XM database will be guided by MKA guidelines and timelines. Data
systems will be developed to ensure automated and ongoing transfer of reports to the MKA system.
Quarterly reports will be drafted and sent to USAID in narrative and presentation format within 30 days of
the end of the quarter. The report format was designed in collaboration with the agreement officer team at
USAID. The quarterly report will provide M-RITE activities and key accomplishments and challenges under
M-RITE Activity Monitoring, Evaluation, and Learning Plan 17
each result area. Along with reporting performance indicators, the report will also report on learning for each
quarter. Finally, additional USAID-requested reports will be written as needed.
DATA ANALYSIS AND USE M-RITE’s data analysis and use approaches and practices aim to serve three main purposes:
1. Track progress toward stated results by comparing observed to intended changes (difference
between target and baseline). We will have a project-wide agreement on performance and a
communication tool to discuss challenges and opportunities to inform adaptive project
management.
2. Support adaptive learning by integrating contextual data and insights with quantitative data so we
can identify why and how we are achieving anticipated results or lagging behind.
3. Identify equity gaps by performing sub-group analyses, particularly by known equity
bottlenecks/dimensions like wealth quintile, to ensure we get closer to our goal of increasing access
to services for zero-dose children and at-risk populations.
STRENGTHENING M-RITE CAPACITY AND CULTURE FOR DATA ANALYSIS
AND USE
As part of Result 3, M-RITE will apply strategies to improve data use in the programs where we work. We will
also apply these strategies to our own data practices to facilitate adaptive learning and improve the overall
effectiveness of M-RITE support. These strategies aim to strengthen capabilities, improve behaviors and
motivations, and create better opportunities through updated systems and processes to use data. M-RITE’s
data system will include built-in dashboards with decision-support cues to make information more available
and actionable. All project staff will be trained to use these tools and will self-assess their data use and
interpretation skills in order for the MEL team to offer capacity strengthening where needed. M-RITE will
explore the use of behavioral nudges to increase analysis, use, and adaptive learning with the team’s
behavioral science and HCD partners.
USING NOVEL ANALYTIC APPROACHES
M-RITE’s emphasis on equity and zero-dose children will require analytic approaches that use meaningful
disaggregation and different measurement approaches.6,7 Performance monitoring indicators will be
disaggregated by variables with known equity dimensions, as noted above, in the project data dashboards
and all reports to ensure we are closing equity gaps. We will measure the IA2030 indicator for geographic
coverage inequality through district-level disaggregation and will track progress on narrowing geographic
inequalities by tracking cohorts of low-coverage districts over time. We will use district-level data to
identify districts with the greatest number or proportion of zero-dose children to initially identify where
M-RITE should work. Because neither survey nor administrative data offer a complete set of variables
related to equity, we will triangulate both when feasible and look for proxy indicators in existing or novel
data collection platforms.8 Where child-level data are available (e.g., through EIRs), we will leverage them
for more granular and real-time monitoring of performance indicators and their equity disaggregation.9
When available, we will use EIR data to measure immunization timeliness during and after COVID-19 and
to inform efforts to identify and catch-up missed children.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 18
Our approach to equity analysis and disaggregation extends to process indicators in addition to outcome
indicators; for example, disaggregating service readiness indicators such as vaccine supply availability by
rural/urban and facility type as part of our focus on improving the equitable distribution of vaccines and
supplies. Where sex disaggregated data are available, we will use them to measure gender inequity in
coverage rates, particularly in countries that have historical evidence of coverage differences between girl
and boy children; or where these differences have emerged during COVID-19 (e.g., in Pakistan).
COLLABORATING, LEARNING, AND ADAPTING All aspects of M-RITE’s MEL and implementation approaches align with USAID’s CLA practices to inform
project strategy and implementation to accelerate sustainable progress toward improved immunization
coverage and equity. In the context of COVID-19, CLA is more important than ever as it can help M-RITE and
its stakeholders prepare for and respond to shifting context, priorities, resources, and partners. M-RITE uses
CLA and adaptive learning practices and processes along the program cycle, as illustrated in Figure 3.
FIGURE 3: SIX ESSENTIAL PROCESSES ENABLE ADAPTIVE LEARNING AT THE PROJECT AND
ORGANIZATIONAL LEVELS, FROM MKA’S DRAFT ADAPTIVE LEARNING TOOLKIT
During activity design, M-RITE uses co-creation, supports use of evidence and data, and enables evaluative
thinking to identify critical assumptions. For example, the M-RITE MEL team will support activity leads to
use causal link monitoring as part of activity design, particularly for activities that involve many
assumptions or for which the evidence base is weaker.4
During routine monitoring, our choice of indicators and data management systems ensures that the right
data are collected and presented at the right time for decisions on program strategy, scale up, and adapting
to other locations and contexts. These data and insights will be discussed in quarterly data review and action
meetings with MEL and activity staff. M-RITE will use these meetings to assess progress and situation, adjust
or adapt strategies and priorities, reflect on learning questions, and propose alternatives to achieve better
results. We will encourage participants at all levels to reflect on qualitative information and consider the
interrelationships, perspectives, and boundaries within the system that may affect implementation and how
these affect the proposed approaches and questions.
Other global, regional, and national learning fora will include pause and reflect sessions, learning cafes, and
virtual conferences to share real-time as well as synthesized insights, emerging evidence, and knowledge. As
M-RITE begins to work in multiple countries, we will identify opportunities for cross-country knowledge
sharing and learning.
M-RITE will communicate results from knowledge management and communication activities through a
variety of activities.
LEARNING AGENDA AND EVALUATION The primary objective of M-RITE’s learning agenda is to improve program design and implementation
through ongoing analysis and reflection on key questions. The aims of M-RITE’s learning agenda mirror
M-RITE Activity Monitoring, Evaluation, and Learning Plan 19
those of the XM learning agenda:
● Capture relevant, prioritized learning that cuts across the suite of awards.
● Test and explore assumptions and hypotheses laid forth by the M-RITE and XM TOC.
● Fill knowledge gaps critical for the MOMENTUM suite to make informed decisions, including
course corrections.
● Generate evidence for MOMENTUM and its partners.
● Help fill gaps in knowledge important to the USAID portfolio and to the larger body of
stakeholders at country and global levels.
Alignment with MKA’s aims and specific learning questions will facilitate XM learning. M-RITE will
participate in the XM MEL working group and learning sub-group to share learnings and coordinate efforts
to gather and synthesize information. M-RITE will work with Gavi to identify concrete opportunities to
partner on specific learning questions and lessons for applying an adaptive learning approach more
broadly.
Our learning questions are organized under the following themes:
● Equity (including zero-dose/under-immunized).
● COVID-19.
● Health systems, resilience, and life-course/integration. ● Demand, trust, and community. ● Co-creation/HCD/adaptive learning/innovation. ● Capacity strengthening/human resources. ● Partnerships.
The full list of learning questions is in Appendix C. M-RITE’s learning agenda includes: 1) externally relevant questions that aim to fill some of the most important knowledge gaps in immunization today, and 2) internally relevant questions to assess M-RITE’s contributions and the validity of the underlying M-RITE and XM TOC. M-RITE’s learning agenda reflects our consortium’s expertise and considers where we can contribute best to global, regional, and national conversations related to evidence.
We anticipate answering many of the external-facing questions through workplan activities using
approaches such as systematic and realist reviews and triangulation of secondary data. For internal-facing
questions, we will largely use causal link monitoring and other process evaluation tools embedded in case
studies to understand why and how a given project strategy works. The MEL team will adapt existing tools
such as ‘most significant change’ and realist frameworks to synthesize emerging learnings. M-RITE will
discuss and reflect on insights, evidence, and knowledge during the learning fora listed in the CLA section.
EVALUATION
We define evaluation as a systematic way to generate evidence related to whether, how, and why
something works. M-RITE’s approach to adaptive learning draws on many approaches from
developmental and process evaluation, noting that a range of approaches can be used to generate
evidence for decision-making. However, we will transition to a more traditional evaluation mindset for
certain learning questions and evaluation topics when the evidence threshold to move forward with the
decision in question is high (e.g., there is extensive uncertainty about an intervention), or there is a high
risk of bias with less systematic learning approaches. We will identify these priorities and needs with key
M-RITE Activity Monitoring, Evaluation, and Learning Plan 20
stakeholders and propose developmental evaluation approaches where possible to strengthen adaptive
learning capacity in countries.
M-RITE anticipates that the evaluation of innovations will be a priority for many stakeholders, and where
feasible we will propose quasi-experimental approaches to evaluating innovations as a less expensive
alternative to experimental designs. We will draw from the discipline of process evaluation and recent
advances in the evaluation of complex interventions and/or complex systems to understand why and how
something is effective or not. As needed, we will incorporate economic evaluation to support countries’
Journeys to Self-Reliance.
MEL ROLES AND RESPONSIBILITIES The MEL advisor, working closely with other MEL staff as appropriate, will be responsible for AMELP
implementation. Within the project, s/he will work in partnership with the project senior management team,
including the MEL, immunization technical, and results leads; senior knowledge management strategist;
project director; and team members. The MEL lead will engage with MKA MEL team members, project
stakeholders, and USAID to implement the AMELP across all project activities while responding to activity
learning and changes in USAID priorities, strategies, and external events.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 21
APPENDIX A. CONTEXTUAL INDICATORS
Indicator Data source Disaggregation Reporting
frequency
Demographic
Total population World Bank WDI Total Annual
Total births World Bank WDI Total Annual
Mortality World Bank WDI Total Annual
Number under-5 child deaths WHO Age, sex Annual
Under-5 mortality rate WHO Age, sex Annual
Number neonatal deaths WHO Age, sex Annual
Neonatal mortality rate WHO Age, sex Annual
Number infant deaths WHO Age, sex Annual
Infant mortality rate WHO Age, sex Annual
COVID-19 DATA
M-RITE Activity Monitoring, Evaluation, and Learning Plan 22
# of COVID-19 cases in previous three months HMIS/COVID-19 tracking
(secondary data)
N/A Quarterly
# of COVID-19-related deaths in previous three
months
HMIS/COVID-19 tracking
(secondary data)
N/A Quarterly
# of COVID-19 cases in health workers in previous
three months
HMIS/COVID-19 tracking
(secondary data)
Cadre Quarterly
# of COVID-19-related deaths among health
workers in previous three months
HMIS/COVID-19 tracking
(secondary data)
Cadre Quarterly
COVID-19 policy stringency score Oxford COVID-19
Government Response
Tracker
N/A Annual
National recommendations related to the delivery
of immunization services during COVID-19
COVID-19 Policy Tracker N/A Annual
Reach
Number of districts with M-RITE support for EPI
programs
Program records N/A Annual
Total population in areas with M-RITE support for
EPI programs
Estimate/program
records
N/A Annual
M-RITE Activity Monitoring, Evaluation, and Learning Plan 23
Number of health facilities with M-RITE support
for EPI programs
Program records Type of SDP Annual
Number of organizations receiving M-RITE support Program records Type of organization (government,
parastatal, private [CSO, NGO, for-
profit]), type of USAID partner (new,
existing)
Annual
Context
Number of instances of stresses and shocks in [1]
M-RITE-supported areas
Program records Type of stress or shock Quarterly
%/# of M-RITE-supported health facilities with no
[2] stockouts of any tracer stock for vaccine in the
past three months
HMIS/LMIS/survey/
program records
Type of SDP; urban/rural;
numerator/denominator
Quarterly
Distance from nearest health facility DHS Urban/rural; sub-national unit; wealth
quintile; maternal education
Annual
Number of measles cases VPD surveillance Sub-national unit Annual
Gender and equity
% of women who report shared or full ability to
have the final say in decisions about their child
DHS Urban/rural; sub-national unit; wealth
quintile; maternal education
Annual
M-RITE Activity Monitoring, Evaluation, and Learning Plan 24
Difference in DTP3 coverage by wealth, maternal
education, and rural/urban
DHS Annual
Service organization and health systems
% of vaccine doses that are delivered through
fixed-sites (e.g., facility) versus outreach-based
services
Country multi-year plan Facility, outreach-based services Annual
Daily vaccination clinics available? Country multi-year plan Type of SDP, if available Annual
Number of antigens in the national EPI schedule Joint Reporting Form N/A Annual
% of health budget spent on immunization JRF N/A
% of health care providers who received a
supervision visit in the past quarter
SPA/SARA Type of provider; SDP type; sub-
national unit; provider gender
Annual
% of health care providers who were trained on
immunization safety in last 2 years
SPA/SARA Type of provider; SDP type; sub-
national unit; provider gender
Annual
Nurses per capita World Bank WDI Annual
Demand and community
M-RITE Activity Monitoring, Evaluation, and Learning Plan 25
% of caregivers with knowledge about vaccines
and the recommended schedule
DHS Urban/rural; sub-national unit; wealth
quintile; maternal education;
maternal parity
Annual
% of caregivers who trust the safety and efficacy
of vaccines
DHS Urban/rural; sub-national unit; wealth
quintile; maternal education;
maternal parity
Annual
% of children whose mothers intend to vaccinate
their child
DHS Urban/rural; sub-national unit; wealth
quintile; maternal education;
maternal parity; child sex
Annual
M-RITE Activity Monitoring, Evaluation, and Learning Plan 26
APPENDIX B. PRIORITIZED PERFORMANCE INDICATOR
REPORTING SHEETS
Indicator 1.0.1 Dropout between first dose (DTP1) and third dose (DTP3) of DTP-containing vaccines in MOMENTUM-supported areas
Rationale XM core indicator. Recommended by IA2030. This is an established indicator that is one measure of the construct of "under-immunization" and a proxy measure system accessibility and service quality.
Precise definition The difference between the number of percent of children<12 months who
received DTP1, but not DTP3 vaccine.
Data source(s) HMIS
Data disaggregation SDP, country, district, urban/rural
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads
Indicator 1.0.2 Number or percent of children <12 months who received DPT3/Penta 3 vaccine in M-RITE-supported areas
Rationale
This is an established indicator for vaccination coverage, included in IA2030 and XM lists. M-RITE acknowledges that while it is the most commonly used indicator for coverage, it does not represent full coverage of all antigens, nor does it reflect effective coverage of DPT-containing vaccines.
Precise definition Number/percent of children <12 months who received DPT3/Penta 3 vaccine
in M-RITE-supported areas.
Data source(s) HMIS
Data disaggregation SDP, country, district, urban/rural
M-RITE Activity Monitoring, Evaluation, and Learning Plan 27
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads
Indicator 1.0.3 Number or percent of surviving infants who received 1st dose of measles-containing vaccine in M-RITE-supported areas
Rationale XM core indicator begins to measure vaccination coverage across the life-course as 1st dose of measles-containing vaccine is administered at 9 months of age.
Precise definition Number/percent of surviving infants who received 1st dose of measles-
containing vaccine in M-RITE-supported areas.
Data source(s) HMIS
Data disaggregation SDP, country, district, urban/rural
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads
Indicator 1.0.4 Number or percent of surviving infants who received 2nd dose of measles-containing vaccine (MCV2) in M-RITE-supported areas
Rationale XM core indicator measures vaccination coverage across the life-course as MCV2 is administered at 18 months of age.
Precise definition Number/percent of surviving infants in previous year’s birth cohort who
received MCV2 in M-RITE-supported areas.
Data source(s) HMIS
M-RITE Activity Monitoring, Evaluation, and Learning Plan 28
Data disaggregation SDP, country, district, urban/rural
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads
Indicator 1.0.5 Number or percent of surviving infants who did not receive 1st dose of DPT/Penta vaccine in M-RITE-supported areas
Rationale
This is the definition of zero-dose children used by IA2030. It aims to measure how many children did not receive the first dose of a DTP3-containing vaccine. It could indeed be the case that they received birth doses of BCG or HepB, or oral polio. It is calculated by subtracting the # of children who received DTP1 from the estimated surviving birth cohort to arrive at the number who did not receive it.
Precise definition Number/Percent of surviving infants who did not receive 1st dose of
DPT/penta vaccine (DTP1) in M-RITE -supported areas
Data source(s) HMIS
Data disaggregation SDP, country, district, urban/rural
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads
Indicator 1.0.7 Number or Percent of live births who received a birth dose of BCG (BCG0) in M-RITE-supported areas.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 29
Rationale
This indicator measures the administration of BCG vaccine within the first 24 hours of a newborn’s life. Because it is delivered in maternity wards, it is not traditionally the responsibility of the EPI program, but it is important to measure and track for advocacy purposes. M-RITE will not be held accountable for performance on this indicator.
Precise definition
Numerator: Number of newborns receiving BCG in the first 24 hours of life
Denominator: Number of live births
Definitions may vary by country as this is a less-established routine M&E
indicator in many national HMIS systems.
Data source(s) HMIS
Data disaggregation SDP, country, district, urban/rural
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads
Indicator 1.0.8 Equity DTP3 coverage among historically unreached communities or group
Rationale This tailored indicator will measure changes over time in DTP3 coverage among communities or groups that M-RITE identifies as being under-served, socially marginalized, or at-risk of being under-vaccinated for other reasons.
Precise definition
This indicator will be tailored to national and sub-national context and
“communities” or “groups” will depend on a specific national or sub-national
drivers of under immunization or zero-dose.
Data source(s) HMIS, Survey depending on country
Data disaggregation Country, district, urban/rural
M-RITE Activity Monitoring, Evaluation, and Learning Plan 30
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads.
Indicator 1.1.1 Percent of M-RITE-supported districts (or facilities) reporting full stock availability at service delivery level during re-supply cycle or reporting period
Rationale Stockout is a common driver of under-vaccination, so this indicator will identify areas in need of supply chain related support, and measure progress in stock availability over time.
Precise definition Full stock availability is defined as the absence of any stock-out for vaccines
and supplies.
Data source(s) LMIS, HMIS, or other stock databases
Data disaggregation Country, district
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads.
Indicator 1.1.3 Percent of health facilities with evidence of utilization of an up-to-date and quality microplan to identify at-risk or vulnerable communities and children
Rationale
A microplan defines how to reach clients, how many people should be targeted for services in the area, and how frequently high-quality services are provided. A microplan is developed by all stakeholders at each level. An effective micro-plan will support health facilities and district teams to: 1) identify target population; 2) design data and graphic mapping; 3) prioritize plans to reach target population; 4) define realistic actions; 5) reduce inequity and improve the quality of immunization services.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 31
Precise definition
"Up-to-date" will be defined according to the national recommendations and context. “At-risk” will be defined according to national and sub-national context and M-RITE’s assessment.
Data source(s) Project records
Data disaggregation Country, district, SDP type, public/private
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads.
Indicator 1.1.6 Percent of health workers in M-RITE-supported facilities who can explain how to identify and reach zero-dose and under-immunized children.
Rationale
This indicator measures health worker capabilities related to the specific behaviors needed to identify and reach zero-dose children. In some countries we may tailor it to measure direct observation of the application of these capabilities (e.g. by observing their use of facility register data to identify which communities need outreach). These capabilities are a necessary but not sufficient condition to actually reaching zero-dose and underimmunized children, yet important and measurable steps in the results chain.
Precise definition Percent of health workers in M-RITE-supported facilities who can explain how to identify and reach zero-dose and under-immunized children.
Data source(s) Project records
Data disaggregation Country, district, SDP type, public/private
M-RITE Activity Monitoring, Evaluation, and Learning Plan 32
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads.
Indicator 1.1.7 Percent of M-RITE-supported health facilities who improve their readiness to reach zero-dose and under-immunized children with high-quality immunization services. (T)
Rationale
Building from Gavi’s theory of change to reduce the number of zero-dose children, M-RITE recognizes the important step of reaching zero-dose children once they have been identified. Through M-RITE’s baseline assessments, we will support efforts to define the behaviors and actions of health care workers, as well as necessary processes, to reach zero-dose children and convert those into a readiness checklist. These may include constructs such as having an up-to-date microplan, implementing community engagement activities, and having sufficient operational budget for outreach activities.
Precise definition “Readiness” will be defined in country PMPs based on needs identified through M-RITE’s baseline assessment.
Data source(s) Project records
Data disaggregation Country, district, SDP type, public/private
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 33
Indicator 1.3.1 Percent of M-RITE-supported areas with activities led by the EPI program designed and implemented to overcome specific vaccine demand- or utilization-related barriers to immunization in high-risk communities.*
Rationale
This aligns with the IA2030 indicator to measure actions to overcome demand- and utilization-related barriers. When providing in-country TA, M-RITE will assess the baseline status of activities to overcome demand- and utilization-related barriers and determine if current activities and plans are sufficient, or require updating.
Precise definition
"High risk" will be defined at the level of planning and implementation, ideally district or facility through M-RITE’s co-creation process.
Data source(s) Project records
Data disaggregation Country, district, type of SDP (as relevant)
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads.
Indicator 2.0.2 Percent of national and sub-national immunization stakeholders whose capacity* was strengthened** by M-RITE-supported capacity strengthening activities. (T)
Rationale
This indicator will be further updated and tailored to reflect specific capacity building needs in countries where M-RITE works. This indicator measures the initial outcomes of M-RITE support capacity strengthening activities, such as trainings, coaching, peer-learning, etc.
Precise definition
Numerator: National and sub-national immunization stakeholders whose capacity was strengthened by M-RITE support capacity strengthening activities, as measured by a post-test, direct observation, or other feasible and appropriate approaches
M-RITE Activity Monitoring, Evaluation, and Learning Plan 34
Denominator: National and sub-national immunization stakeholders who participated in M-RITE supported capacity strengthening activities Capacity building interventions are defined as any forms of interventions or activities that improve the ability of a person, group, or organization to meet objectives or to perform better. We will tailor the definition of “capacity,” but it may include skills, knowledge, behaviors, or actions taken.
Data source(s) Project records
Data disaggregation Country, stakeholder type, gender of stakeholder
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads.
Indicator 2.2.2 Number of global, national, and sub-national policies (including strategies, guidelines, and operational plans) developed with M-RITE support related to immunization services or MOMENTUM cross-cutting areas.
Rationale This indicator measures the initial outcomes of M-RITE's technical assistance toward improved policy for immunization.
Precise definition
Number of global, national, and sub-national policies (including strategies,
guidelines, and operational plans) developed with M-RITE support related to
immunization services or MOMENTUM cross-cutting areas.
Data source(s) Project records
Data disaggregation Country, type of policy, topic (e.g., COVID-19 vs. non-COVID-19)
Data collection and reporting frequency
Annual
PY1 target 1 (DRC)
M-RITE Activity Monitoring, Evaluation, and Learning Plan 35
Individual responsible for reporting data
Reporting by result leads and in-country M&E leads.
Indicator 2.3.1 Percent of M-RITE-assisted local organizations with improved performance
Rationale This indicator measures whether USG-funded capacity-development efforts have improved performance in supported organizations.
Precise definition
This is the USG indicator CBLD-9:
https://www.usaid.gov/sites/default/files/documents/cbld-
9_pirs_final_fy20_10.1.2020.pdf
Capacity is the ability of people, organizations, and society as a whole to
manage their affairs. Capacity development is the process of strengthening
and maintaining such capacity. Capacity is a form of potential; it is not visible
until it is used. Therefore, performance is the key consideration in
determining whether capacity has changed. Organizations with improved
performance will have undergone a deliberate process to improve execution
of organizational mandates to deliver results its stakeholders.
M-RITE will measure improvement organization using already existing tool
such as university accreditation process, ISO professional farmer
organization guidelines, OCA tool, etc.) and for monitoring and measuring
changes on a key performance metric (e.g., increased sales, reduced staff
turnover, increased efficiency of service delivery, etc.).
Data source(s) Project records
Data disaggregation Country, type of organization
Data collection and reporting frequency
Annual
PY1 target TBD upon country buy-in
Individual responsible for reporting data
Reporting by in-country M&E leads.
Indicator 3.0.1 Number of evidence products, tools, or data analyses produced by M-RITE that include actionable recommendations and an equity and gender equity lens.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 36
Rationale
This indicator measures M-RITE’s evidence outputs. This indicator is worded to ensure that M-RITE produces evidence outputs that are more likely to be used (e.g. by including actionable recommendations) and that will accelerate the achievement of equitable immunization (e.g. by including an equity and gender equity lens).
Precise definition
The term "actionable recommendations" is measured using a checklist from
FAO. We will assess equity and gender lens using our M-RITE process
indicators related to pro-equity/gender-sensitive activity design and
implementation.
Data source(s) Project records
Data disaggregation Topic (e.g., COVID-19 or non-COVID-19), country
Data collection and reporting frequency
Semi-annual
PY1 target 9
Individual responsible for reporting data
Reporting by each result lead and in-country M&E leads.
Indicator 3.2.2 Number of individuals reached through M-RITE co-creation, dissemination, knowledge exchange, and knowledge translation activities and events.
Rationale This indicator measures reach of M-RITE knowledge products and knowledge translation/dissemination activities.
Precise definition
Individuals are counted in "reach" if they were the active participant in an M-RITE co-creation or knowledge translation/dissemination-related event or activity. Active participation could be defined as those people who actively have to choose to engage in content - by attending meetings, opening e-mails, logging into webinars, etc.
Data source(s) Project records
Data disaggregation Organization, gender (where possible), country
Data collection and reporting frequency
Semi-annual
M-RITE Activity Monitoring, Evaluation, and Learning Plan 37
PY1 target 500
Individual responsible for reporting data
Reporting by each result lead and in-country M&E leads.
Indicator 3.3.1 Number of innovations supported by M-RITE assistance.
Rationale
Particularly during COVID-19, innovations in products, processes, tools, approaches, and service delivery models are needed to maintain immunization services. M-RITE will contribute to the identification and implementation of these innovations and track this output.
Precise definition
‘Innovations’ are products, processes, tools, approaches, service delivery
models, and/or other interventions (broadly defined) that have the potential
to achieve significant (not incremental) improvements in development
outcomes versus existing alternatives and are intended to improve the lives
of ultimate beneficiaries. Here, innovations are defined not by their novelty,
but by their potential to achieve significant improvements in development
outcomes versus existing alternatives. (From STIR10 PIRS.) "Supported"
refers to technical, in-kind, or financial support from M-RITE related to the
introduction, testing, or adoption of innovations.
Data source(s) Project records
Data disaggregation Country, topic, new or ongoing support
Data collection and reporting frequency
Annual
PY1 target 2
Individual responsible for reporting data
Reporting by each result lead and in-country M&E leads.
Indicator 3.4.2 Number of global and regional documents (guidelines, strategies, tools, etc.) that M-RITE contributes to.
Rationale This indicator measures M-RITE's active participation and thought leadership in global and regional technical conversations.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 38
Precise definition Number of global and regional documents (guidelines, strategies, tools, etc.)
that M-RITE contributes to.
Data source(s) Project records
Data disaggregation Topic (e.g., COVID-19 or non-COVID-19), new or ongoing
Data collection and reporting frequency
Semi-annual
PY1 target 4
Individual responsible for reporting data
Reporting by each result lead and in-country M&E leads.
Indicator 4.0.2 Number of multi-sectoral collaborations formed or strengthened at global, regional, national, and sub-national levels as a result of M-RITE support to address immunization coverage and equity issues.
Rationale This indicator measures M-RITE's contributions to forming or strengthening multi-sectoral collaborations, a necessary step in the causal chain toward improved immunization coverage and equity.
Precise definition
We define "collaboration" as a relationship between two organizations
working together with the goal of jointly producing an output or
contributing to a common goal. In this case, a multi-sectoral collaboration
must occur between two organizations from different sectors (e.g.,
education and health, finance and health). We will define "formation" as a
new relationship, and "strengthening" as a new goal on an existing
collaboration relationship.
Data source(s) Project records
Data disaggregation Collaboration type, topic, country, new (formed) or existing (strengthened)
Data collection and reporting frequency
Annual
PY1 target 2
M-RITE Activity Monitoring, Evaluation, and Learning Plan 39
Individual responsible for reporting data
Reporting by each result lead and in-country M&E leads.
Indicator 4.0.3 Number of partnerships formed or strengthened at global, regional, national, and sub-national levels to address immunization coverage and equity issues.
Rationale This indicator measures M-RITE's contributions to forming or strengthening multi-sectoral partnerships, a necessary step in the causal chain toward improved immunization coverage and equity.
Precise definition
We will define "partnership formation" during the partnership landscaping activity and based on existing concepts that partnerships can be organic or mandated, but typically share common goals. We will measure partnership strength at baseline and end line of related activities using a validated partnership survey tool that measures perceived effectiveness, efficiency, and legitimacy of the partnership, as well as its composition.
Data source(s) Project records
Data disaggregation Partnership type, topic, country, new (formed) or existing (strengthened)
Data collection and reporting frequency
Annual
PY1 target 1
Individual responsible for reporting data
Reporting by each result lead and in-country M&E leads.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 40
APPENDIX C. ILLUSTRATIVE LEARNING AGENDA THEMES AND
QUESTIONS
Theme Proposed learning questions to be answered over M-RITE life-of-project (PY1 focus questions indicated).
1. Equity (including
zero-dose, under-
immunized,
gender)
1.1 What are the information-related obstacles and opportunities to identifying, targeting, and reaching zero-dose and under-immunized children? What are promising information systems, tools, or practices to help with identifying, targeting, and reaching zero-dose and under-immunized children? (Similar to Gavi learning question 1.6: What are the evidence gaps at national/sub-national level to monitor and measure for zero-dose and have Gavi data investments contributed to identifying and quantifying zero-dose children and missed communities?) [PY1]
1.2 What strategies can overcome barriers to adequate financing that most significantly affect zero-dose children? [PY1]
1.3 How can improved governance increase vaccination among zero-dose and under-immunized children? [PY1]
1.4 How can supply chain interventions improve the equity of immunization coverage, specifically by overcoming obstacles to reaching zero-dose children (e.g., geographic access barriers among rural/remote communities)? [PY1]
1.5 Where and who are zero-dose children, and missed communities? Why are they being missed? (From Gavi learning agenda question 1.1)
1.6 What are the most effective and cost-effective strategies to overcoming entrenched obstacles to reaching zero-dose children? Under-immunized children?
1.7 What strategies did M-RITE use to improve immunization equity? How did approaches mitigate inequities in demand and access? (Aligned with XM learning question)
1.8 What are some of the most relevant innovations for improving identification of, and targeting, and reaching zero-dose and under-vaccinated children? How did M-RITE leverage these innovations? (Aligned with Gavi learning question 5.1: How do Gavi’s investments in innovation contribute to improved immunization outcomes? Incl. innovation in digitization of data and monitoring of zero-dose children and missed communities? And enable identification and scaling of these across products, services, and practices?)
1.9 What is the cost-effectiveness and effectiveness of gender-sensitive, -responsive, or -transformative strategies? How can this evidence be used to inform policy change?
1.10 Whether, how, and why do various approaches to micro-planning improve the identification of and targeting, and reaching zero-dose and under-immunized children?
1.11 How did M-RITE achieve its objectives related to improving immunization coverage and equity? (Aligns with XM learning question)
M-RITE Activity Monitoring, Evaluation, and Learning Plan 41
2. COVID-19 2.1 What are the most effective strategies to mitigate the effects of COVID-19 on
the organization, access, and use of RI services in different settings and
contexts? What are the equity implications of these strategies for vulnerable
communities (e.g., rural remote, urban poor, children in conflict settings)?
Which of these strategies or adaptations should be recommended for
ongoing implementation? [PY1] (Aligned with Gavi learning question 11.8:
What approaches were selected and were they effective at re-establishing RI,
and identifying and reaching zero-dose children and missed communities?
Including the role of innovations. Aligned with MKA learning question: How
did COVID-19 affect health systems, how did the response affect
MNCHN/FP/RH services delivery and access, and what was done by
programs/countries to mitigate impact?)
2.2 What can we learn from the introduction of other life-course vaccinations
(e.g., MCV2, RTSS, Td extended schedule, HPV [youth], meningitis A) and
platforms to reach non-standard populations for COVID-19 vaccine
introduction? [PY1]
2.3 How did the COVID-19 pandemic influence short- and longer-term changes to
the financing, planning, organization, delivery, and monitoring of
immunization services? How did it influence client trust, demand, and
satisfaction with immunization services and the health system more broadly?
2.4 How did the contextual shock of the COVID-19 pandemic influence the
implementation of M-RITE and the results it achieved? How did M-RITE adapt
its overall project approach to the COVID-19 context?
2.5 [New in version 2] What are the bottlenecks to effective and equitable
vaccination against COVID-19 at the sub-national level, and what approaches
were effective in overcoming these bottlenecks.
3. Health systems,
resilience, and
life-course/
integration
3.1 What are valid, feasible indicators to measure integration of immunization
and other primary health care (PHC) services?
3.2 How has integration with other PHC services been used to reach zero-dose
children and missed communities? What has worked well, or not, and why?
(Aligns with Gavi learning question 10.4)
3.3 How did M-RITE contribute to strengthening health systems resilience? What
capacity strengthening efforts (whose and what dimensions of capacity) were
successful to strengthen resilience of health systems? (Aligns with XM
learning question)
M-RITE Activity Monitoring, Evaluation, and Learning Plan 42
4. Demand, trust,
community
4.1 What are effective or high-potential partnership models for underserved
communities? How have models shifted during COVID-19? [PY1] (Aligns with
Gavi learning question 1.3: What are effective ways to engage with other
partners to reach the marginalized, missed communities and zero-dose
children (including integration with PHC?)
4.2 What are valid and feasible indicators and data collection approaches to
measure respectful care during an immunization encounter? [PY1]
4.3 What community engagement strategies are most effective at reducing the
number of zero-dose children?
5. Innovation/ HCD/
co-creation/
learning
5.1 What is the added value of a “co-creation” approach to designing and
implementing M-RITE country projects? What does the ideal co-creation
approach entail, and how can it be adapted to different contexts and needs?
[PY1]
5.2 How was active use of data and adaptive learning improved for
immunization? (Includes capacity-building at community, facility,
subnational, and national levels) (Aligned with XM learning question)
5.3 What evidence is there of institutionalization or sustainable change in use of
CLA, adaptive learning, and HCD approaches in MOMENTUM countries?
What contributed to these successes? (Aligned with XM learning question)
5.4 What is the added value of an HCD approach to identifying and overcoming
entrenched obstacles, compared to more traditional approaches? How did
M-RITE’s HCD approach contribute to its achievement of results?
5.5 What are the most effective ways to scale innovations that reduce the
number of zero-dose children? How do we ensure innovation does not
increase inequity?
6. Capacity
strengthening
6.1 What capacity-building strategies/interventions (or combination of
strategies) were effective in increasing capacity at the individual,
organization, community, and system levels? (Aligned with XM learning
question)
6.2 Were any associations seen between strengthened capacity and
improvements in equitable immunization coverage? (Aligned with XM
learning question)
7. Partnerships 7.1 How have partnerships contributed to strengthening immunization
programs, and what is the untapped potential of strategic partnerships for
improving equitable immunization coverage? [PY1]
7.2 How can strategic partnerships contribute to planning, introduction, and
acceptance of COVID-19 vaccine? [PY1]
7.3 What strategic partnerships (and under what conditions) were successful
(feasible, acceptable)? (Aligned with XM learning question)
M-RITE Activity Monitoring, Evaluation, and Learning Plan 43
7.4 Which program strategies were most effective in building or strengthening
cross-sectoral collaboration and partnerships (e.g., health and non-health,
health and education, PPPs, private sector)?
7.5 What is M-RITE’s approach to measuring the contribution of partnerships to
improving immunization equity?
M-RITE Activity Monitoring, Evaluation, and Learning Plan 44
APPENDIX D. SUPPLEMENTARY INDICATORS
Indicator Code
Indicator Name
Indicator Type
Indicator Definition Numerator Denominator Data source (e.g. HMIS)
Indicator source/alignment (e.g. Gavi)
Disaggregation
S1.0.1 Number of programs where M-RITE-supported the development or improvement of evidence-based and funded plans to address vaccine coverage of high-risk communities.
Initial outcome
Number of M-RITE-supported EPI national or subnational programs with evidence-based and funded plan to address vaccine coverage of high risk communities. "High risk" should be defined on a country-by-country basis. When providing in-country TA, M-RITE will assess the current status of equity plans and guidelines, and will assess whether the current equity plan is sufficient, or requires updating. For decentralized countries, the assessment and updating may occur at a lower level (e.g. province, county).
Number of M-RITE-supported national or subnational EPI programs that have an evidence-based and funded plan to address vaccine coverage of high-risk communities.
n/a Project records
IA2030 Country, province or county where applicable
S1.0.2 Percent of M-RITE-supported districts (or facilities) reporting full stock availability at service delivery level during re-supply cycle or reporting period.
Initial outcome
Percent of M-RITE-supported districts (or facilities) reporting full stock availability at service delivery level during re-supply cycle. Full stock availability is defined as the absence of any stock-out for vaccines and supplies.
Number of M-RITE-supported districts (or facilities) reporting full stock availability at service delivery level during re-supply cycle.
Total number of M-RITE-supported districts (or facilities).
LMIS, HMIS, or other stock databases.
IA2030 Country, district, SDP, urban/rural
M-RITE Activity Monitoring, Evaluation, and Learning Plan 45
S1.0.3 Number of programs where M-RITE-supported the development or improvement of policies or standard operating procedures in place to strengthen delivery of immunization services integrated with primary health care, across the life-course.
Initial outcome
Number of M-RITE-supported national or subnational EPI programs with policies or standard operating procedures in place to strengthen delivery of immunization services integrated with primary health care, across the life course. When providing in-country TA, M-RITE will assess the whether sufficient and updated integration policies/SoPs exist.
Number of M-RITE-supported programs with policies or standard operating procedures in place to strengthen delivery of immunization services integrated with primary health care, across the life-course.
n/a Project records
IA2030 district, country
S1.0.4 Percent of health facilities in M-RITE-supported areas that received a high-quality supportive supervision visit for immunization in the last quarter.
Initial outcome
Percent of health facilities in M-RITE-supported areas that received a high-quality supportive supervision visit for immunization in the last quarter. At the outset of a country buy-in M-RITE will use existing checklists to assess the quality of existing supportive supervision visits and through workplan activities will strengthen quality and frequency of supervision, noting where contextual bottlenecks such as inadequate resources to hire supervisors limit progress on this indicator. The timeframe (e.g. "last quarter") may be adjusted according to national standards and norms.
Number of health facilities in M-RITE-supported areas that received a high-quality supportive supervision visit for immunization in the last quarter.
Total number of health facilities in M-RITE-supported areas
Supportive supervision report, rapid telephone facility survey, or health worker interviews.
MCSP process indicators
Country, district, SDP type, urban/rural
S1.0.5 Percent of health workers in M-RITE-supported areas who can explain how to use immunization
Initial outcome
Percent of health facilities in M-RITE-supported areas with up-to-date immunization monitoring charts or other data use tool.
Number of health facilities in M-RITE-supported areas with up-to-date immunization monitoring charts or other data use tool.
Total number of health facilities in M-RITE.
Supportive supervision report, rapid telephone facility survey, or health
MCSP process indicators
Country, district, SDP type, urban/rural
M-RITE Activity Monitoring, Evaluation, and Learning Plan 46
monitoring charts or other data use tools and the meaning of the available information.
worker interviews.
S1.0.6 Percent of clients who report being treated with respect during their last immunization visit in M-RITE-supported facilities.
Initial outcome
Percent of clients who report being treated with respect during their last immunization visit in M-RITE-supported facilities. The definition for "respect" will build from WHO's definition of respectful maternity care, but will be adapted for immunization and validated with clients in DRC.
Number of clients who report being treated with respect during their last immunization visit in M-RITE-supported facilities.
Total number of clients served.
Caregiver exit surveys and rapid phone surveys.
ACCELERATE behavioral indicators
Country, district, SDP, provider type, sex of caregiver
S1.0.7 Percent or number of potential vaccination opportunities that were missed (MOV).
Initial outcome
Percent or number of an individual (child or person of any age) making a contact with health services who is eligible for vaccination (e.g. unvaccinated or partially vaccinated and free of contraindications to vaccination), which does not result in the person receiving one or more of the vaccine doses for which he or she is eligible (WHO).
Missed = Number of individuals (child or person of any age) making a contact with health services who is eligible for vaccination (e.g. unvaccinated or partially vaccinated and free of contraindications to vaccination), which does not result in the person receiving one or more of the vaccine doses for which he or she is eligible.
All opportunities = Number of individuals (child or person of any age) making a contact with health services who is eligible for vaccination (e.g. unvaccinated or partially vaccinated and free of contraindications to vaccination)
MOV surveys (M-RITE will explore more routine data collection approaches)
WHO MOV
M-RITE Activity Monitoring, Evaluation, and Learning Plan 47
S1.0.8 Number of M-RITE-supported districts/provinces that are implementing community accountability mechanisms (e.g. score cards, community consultations, community conversations, community charters, community health committees, civil society hearings etc.) to support women’s, children’s and adolescents’ health, including immunization
Initial outcome
Community accountability mechanisms include score cards, community consultations, community conversations, community charters, community health committees, civil society hearings etc.) to support women’s, children’s and adolescents’ health, including immunization.
Number of M-RITE-supported districts/provinces that are implementing community accountability mechanisms (e.g. score cards, community consultations, community conversations, community charters, community health committees, civil society hearings etc.) to support women’s, children’s and adolescents’ health, including immunization.
n/a. Project records
MCGL District, province, country, type of mechanism
S1.0.9 Percent of health facilities that meet with community members and discuss performance of immunization activities (either by themselves or through participation in broader RI
Initial outcome
Percent of health facilities that meet with community members and discuss performance of immunization activities (either by themselves or through participation in broader RI meetings) in M-RITE-supported areas.
Number of HFs that meet with community members and discuss performance of immunization activities (either by themselves or through participation in broader RI meetings).
Total number of health facilities in M-RITE-supported areas.
Supportive supervision report
MCSP process indicator
Country, district, type of SDP
M-RITE Activity Monitoring, Evaluation, and Learning Plan 48
meetings) in M-RITE-supported areas.
S1.10 Number of M-RITE-supported programs with activities implemented to engage fathers in immunization services.
Initial outcome
Number of M-RITE-supported programs with activities implemented to engage fathers in immunization services.
Number of M-RITE-supported programs with activities implemented to engage fathers in immunization services.
N/A Project Records
Related to proposed ERG gender indicator
country, district, type of activity
S1.11 Proportion of health facilities that received supportive supervision (last quarter) in M-RITE-supported areas.
Initial outcome
Proportion of health facilities that received supportive supervision (last quarter) in M-RITE-supported areas.
Number of health facilities that received supportive supervision (last quarter) in M-RITE.
Total number of health facilities that received supportive supervision (last quarter) in M-RITE-supported areas.
Supportive supervision records
MCSP process indicator
S1.12 Proportion of health facilities with an updated immunization monitoring chart (last month).
Initial outcome
Proportion of health facilities with an updated immunization monitoring chart (last month).
Number of health facilities with an updated immunization monitoring chart (last month).
Total number of health facilities with an updated immunization monitoring chart (last month).
Facility charts
MCSP process indicator
Country, district
M-RITE Activity Monitoring, Evaluation, and Learning Plan 49
S1.13 Proportion of health facilities that met with community members to discuss immunization (last quarter).
Initial outcome
Proportion of health facilities that met with community members to discuss immunization (last quarter).
Number of health facilities that met with community members to discuss immunization (last quarter).
Total number of health facilities that met with community members to discuss immunization (last quarter).
Facility charts
MCSP process indicator
Country, district
S1.14 Proportion of health facilities with at least one qualified and trained vaccine provider (last quarter).
Initial outcome
Proportion of health facilities with at least one qualified and trained vaccine provider (last quarter).
Number of health facilities with at least one qualified and trained vaccine provider (last quarter).
Number of health facilities with at least one qualified and trained vaccine provider (last quarter).
Facility charts
MCSP process indicator
Country, district
S 2.0.1 Number of M-RITE-supported countries that use a reliable method for tracking/monitoring total government routine immunization expenditures at either national or subnational levels.
Initial outcome
Number of M-RITE-supported countries that use a reliable method for tracking/monitoring total routine immunization expenditures at either national or subnational levels. "Reliable method" will be defined during the PY1 activity on this topic, and will include the method that M-RITE produces. Total government routine immunization expenditures comprise the total value of funds from government sources spent on routine vaccines (including any co-financing payments) and immunization-specific system-level costs (e.g., injection safety, immunization supply chain, costs of RI outreach, transport, wages of fully-dedicated immunization staff, etc.) across all administrative levels.
Number of M-RITE-supported countries that use a reliable method for tracking/monitoring total government routine immunization expenditures at either national or subnational levels; depending on direction of work, could be disaggregated by evidence of this process at national or subnational levels (though would need to define if those categories would be mutual exclusive).
n/a Project records
Country, district
M-RITE Activity Monitoring, Evaluation, and Learning Plan 50
S 2.0.2 Number of M-RITE-supported countries with at least one implemented recommendation from a NITAG or other relevant independent technical advisory group.
Outcome Number of M-RITE-supported countries with at least one implemented recommendation from a NITAG or other relevant independent technical advisory group.
Number of M-RITE-supported countries with at least one implemented recommendation from a NITAG or other relevant independent technical advisory group.
n/a Project records
IA2030 Country, district
S 2.0.3 Evidence of increased capacity among immunization program managers in M-RITE-supported programs to plan and implement activities to reach zero-dose and under-immunized children.
Initial outcome
Evidence of increased capacity among immunization program managers at national and sub-national levels in M-RITE-supported programs to plan and implement activities to reach zero-dose and under-immunized children. We will measure management capacity using established tools, adapted to ensure that capacities/competencies related to 'managing for equity' are included.
Number of M-RITE-supported countries and districts with increased capacity to identify and reach zero-dose and under-immunized children.
n/a Project records
n/a Country, district
S 2.0.4 Number of M-RITE-supported countries with legislation in place that is supportive of immunization and commits the government to finance all immunization
Initial outcome
Number of M-RITE-supported countries with legislation in place that is supportive of immunization and commits the government to finance all immunization program functions at all levels.
Number of M-RITE-supported countries with legislation in place that is supportive of immunization and commits the government to finance all immunization programmed
n/a Project records
n/a Country
M-RITE Activity Monitoring, Evaluation, and Learning Plan 51
program functions at all levels.
functions at all levels.
S 2.0.5 Number of improved global, national, and sub-national policies (including strategies, guidelines, and operational plans) developed with M-RITE support that are gender sensitive or gender transformative.
Initial outcome
Number of improved global, national, and sub-national policies (including strategies, guidelines, and operational plans) developed with M-RITE support that are gender sensitive or gender transformative.
Number of improved global, national, and sub-national policies (including strategies, guidelines, and operational plans) developed with M-RITE support that are gender sensitive or gender transformative.
n/a Project records
MCGL Country, district, type of policy
S 2.0.6 Percent of MoH / UNEPI Technical Working Group coordination meetings held where routine immunization was discussed in M-RITE-supported districts.
MoH/UNEPI Technical Working Group coordination meetings are organized by UNEPI and EPI partners to share experiences and challenges in relation to routine immunization to develop strategies for improvement as a country. Routine immunization refers to free immunization services readily available by public and private health facilities as per the immunization schedule. These are different from one-off campaigns aimed at reaching bigger populations and age groups other than the target age of less than one year.
Number of MoH / UNEPI Technical Working Group coordination meetings held with EPI technical partners where routine immunization was discussed.
The number of planned MoH / UNEPI Technical Working Group coordination meetings (the target for UNEPI is to conduct 1
Meeting minutes
MSCP
M-RITE Activity Monitoring, Evaluation, and Learning Plan 52
meeting per month).
S 2.0.7 Number of national level guidelines, manuals, and tools in which concepts are incorporated (IP custom).
This refers to national level / MoH-EPI documents/manuals e.g. Immunization in Practice and operational level training materials in which REC-QI concepts are incorporated.
Number of national level guidelines, manuals, and tools in which concepts are incorporated.
Copies of UNEPI national level documents
MSCP Type of document
S 2.0.8 Percent of MoH / EPI Technical Working Group coordination meetings held where routine immunization was discussed.
Number of MoH / EPI Technical Working Group coordination meetings held with EPI technical partners where routine immunization was discussed.
The number of planned MoH / EPI Technical Working Group coordination meetings (The target for UNEPI is to conduct 1 meeting per month).
Meeting minutes
MSCP N/A
M-RITE Activity Monitoring, Evaluation, and Learning Plan 53
S.3.0.1 Number of M-RITE-supported activities that use an adaptive learning approach to design and implementation.
Output We will assess whether the activities used "adaptive learning" based on the MKA Adaptive Learning toolkit.
Number of M-RITE-supported activities that use an adaptive learning approach to design and implementation.
n/a Project records
MCGL Country, topic
S.3.0.2 Evidence of M-RITE evidence products or data analyses contributing to decisions affecting host country priorities, strategies, programs, policies, or shifts in resource allocations made as a result of MKA capacity building or technical assistance activities.
Outcome We will assess the extent to which M-RITE evidence outputs contribute to decisions using case studies, and applying a validated scale for evidence uptake (Landry).
n/a n/a Case studies MKA Topic (e.g. COVID or non-COVID), country
S.3.0.3 Number of M-RITE-supported review and reflection sessions (e.g. data review meetings, root cause analysis workshops)
Output Number of M-RITE-supported review and reflection sessions (e.g. data review meetings, root cause analysis workshops) where data and evidence are discussed critically.
Number of M-RITE-supported review and reflection sessions (e.g. data review meetings, root cause analysis workshops) where data and evidence
n/a Project records
Similar to MCGL
Country, SDP type
M-RITE Activity Monitoring, Evaluation, and Learning Plan 54
where data and evidence are discussed critically.
are discussed critically.
S.3.0.4 Number of national and sub-national EPI programs that developed or implemented a data improvement plan with M-RITE support.
Output A "data improvement plan" should reflect the most recent WHO, CDC, or Gavi guidance.
Number of national and sub-national EPI programs that developed or implemented a data improvement plan with M-RITE support.
n/a Project records
IA2030 Country
S.3.0.5 Number of national and sub-national EPI programs that strengthened health information systems with M-RITE support to improve the identification of zero-dose and/or under-immunized children.
Output We will define the requirements of systems to identify and reach zero-dose children in the information ecosystem landscaping activity.
Number of national and sub-national EPI programs that strengthened health information systems with M-RITE support to improve the identification of zero-dose and/or under-immunized children.
n/a Project records
n/a Country
M-RITE Activity Monitoring, Evaluation, and Learning Plan 55
S.3.0.6 Evidence of strengthened skills and capacity of immunization stakeholders at national and sub-national levels to take action on equity-related data.
Initial outcome
We will measure skills to take action on equity-related data using an M-RITE checklist adapted from MEASURE tools, but adapted for equity-related data use actions.
n/a n/a Case studies (interviews or surveys with health workers and managers to measure skills and capacity, direct observation of data review and planning meetings to observe skills and capacity)
n/a n/a
S.3.0.7 Evidence of strengthened systems and processes at global, regional, national, and sub-national levels to produce, make available, and take action on equity-related data.
Initial outcome
Systems and processes include technological and management systems such as data review meetings, the use of data entry clerks, the use of dashboards or other decision-support tools, incentives to use data, etc.
n/a n/a Project records
n/a n/a
S.3.0.8 Number of unique visitors to M-RITE web page on MOMENTUM website.
Initial outcome
Number of unique visitors to M-RITE web page on MOMENTUM website.
Number of unique visitors to M-RITE web page on MOMENTUM website.
n/a Project records
Similar to MKA
Country
M-RITE Activity Monitoring, Evaluation, and Learning Plan 56
S.3.0.9 Number of global and regional technical working groups that M-RITE contributes to.
Output Number of global and regional technical working groups that M-RITE contributes to.
Number of global and regional technical working groups that M-RITE contributes to.
n/a Project records
n/a Topic (e.g. COVID or non-COVID)
S.3.10 Number of individuals from M-RITE-supported country programs who contribute to global and regional working groups with M-RITE support.
Output Contribution could be in various ways that include document review, participation in working group meetings, and other events.
Number of individuals from M-RITE-supported programs who contribute to global and regional working groups with M-RITE support.
n/a Project records
n/a Topic (e.g. COVID or non-COVID)
S.4.0.1 Number of new, high-potential, non-immunization organizations engaged to address immunization coverage and equity.
Initial outcome
"New partners" will be defined during the stakeholder landscaping/framework activity. The purpose of this indicator is to measure M-RITE's steps to facilitate the entrance of new, high-potential partners to solve entrenched obstacles related to immunization coverage and equity.
Number of new, high-potential, non-immunization organizations engaged to address immunization coverage and equity.
n/a Project records
n/a Stakeholder type
S.4.0.2 Number of core immunization partners engaged in the planning, design, and implementation of M-RITE
Initial outcome
"Core partners" will be defined during the stakeholder landscaping/framework activity, but are likely to include WHO, UNICEF, Gavi, and CDC. "Co-creation" is defined as a participatory, engaging process that leverages the unique voices of all stakeholders. The purpose of this indicator is to measure M-RITE engagement of these core partners in our activities, which will
Number of core immunization partners engaged in the planning, design, and implementation of M-RITE activities through a co-creation approach.
n/a Project records
n/a Stakeholder type
M-RITE Activity Monitoring, Evaluation, and Learning Plan 57
activities through a co-creation approach.
increase the relevance and usefulness of our outputs.
M-RITE Activity Monitoring, Evaluation, and Learning Plan 59
REFERENCES
1 USAID. Series 200 References. Last updated October 28, 2020. https://www.usaid.gov/who-we-are/agency-policy/series-200/references-chapter
2 USAID. Evaluation Policy. Last updated May 09, 2020. https://www.usaid.gov/evaluation/policy 3 USAID Learning Lab. Collaborating, Learning, and Adapting Toolkit. https://usaidlearninglab.org/cla-toolkit 4 Britt, Heather, Richard Hummelbrunner, and Jacqueline Greene. 2017. Causal Link Monitoring.
http://www.betterevaluation.org/resources/overview/Causal_Link_Monitoring 5 USAID. Maternal and Child Survival Program Mozambique Program Brief Addressing the Denominator
Conundrum for Maternal and Child Health Programs: A New Methodology. https://www.mcsprogram.org/resource-search-results/?_sfm_resource_country=mozambique&_sfm_resource_language_2=english
6 Arsenault, Catherine, et al. 2016. “An Equity Dashboard to Monitor Vaccination Coverage.” Bulletin of the World Health Organization vol. 95,2:128-134. doi:10.2471/BLT.16.178079
7 Restrepo-Mendez, Maria Clara, Godwin Mindra, Alyssa Sharkey, and Richard Duncan. Are the Current Equity Metrics Used by Immunization Programmes Adequate? Equity Reference Group for Immunization. May 2018.
8 Victoria, Cesar and Tove Ryman. March 2018. Potential Approaches to Better Measure and Track Equity in Immunization Using Survey and Administrative Data, and Data Triangulation. Equity Reference Group for Immunization
9 Dolan, Samantha, et. al. March 2019. “Redefining Vaccination Coverage and Timeliness Measures Using Electronic Immunization Registry Data in Low- and Middle-Income Countries.” Vaccine vol. 37, issue 19: 1859-1867. https://doi.org/10.1016/j.vaccine.2019.02.017