46
Monitoring and Evaluation Plan MAMPU: Empowering Indonesian Women for Poverty Reduction August 22 2013 Prepared for: AusAID

MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

Embed Size (px)

Citation preview

Page 1: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

Monitoring and Evaluation Plan

MAMPU: Empowering Indonesian Womenfor Poverty Reduction

August 22 2013

Prepared for:AusAID

Page 2: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan i

TABLE OF CONTENTS Acronyms and Abbreviations ............................................................................................................. iii

1. Introduction ................................................................................................................................... 1

1.1 Why have an M&E Plan? ........................................................................................................ 1

1.2 Principles underlying this M&E Plan ....................................................................................... 1

1.3 Who is this M&E Plan for? ...................................................................................................... 2

1.4 Organisation of this document ................................................................................................ 3

2. What do we want to monitor and evaluate? ............................................................................... 4

2.1 Program Theory ...................................................................................................................... 4

2.2 Evaluability Assessment ......................................................................................................... 5

3. Monitoring Strategy ...................................................................................................................... 6

3.1 What is ‘Monitoring’? ............................................................................................................... 6

3.2 Partner Project Monitoring ...................................................................................................... 6

3.3 Supporting Partners to Monitor Performance ......................................................................... 8

3.4 Program level Monitoring ...................................................................................................... 11

4. A Strategy to Evaluate Results .................................................................................................. 13

4.1 What is ‘Evaluation’? ............................................................................................................. 13

4.2 Key Evaluation Questions (KEQs) ........................................................................................ 13

4.3 KEQ 1: How and to what extent has the program affected the partners and networks’ capacity to influence government reform? ........................................................................................ 14

4.4 KEQ 2: How and to what extent have the partners and networks influenced government reform in relation to the needs and priorities for poor women? ........................................................ 16

4.5 KEQ 3: How and to what extent has the program contributed to improved access and livelihood for poor women? ............................................................................................................... 17

4.6 Special Research Studies ..................................................................................................... 17

4.7 Whole-of-Program Performance Analysis ............................................................................. 18

5. Managing and Reporting M&E information .............................................................................. 19

5.1 Routine Reporting Products .................................................................................................. 19

5.2 Program Completion Report ................................................................................................. 19

5.3 Key meetings......................................................................................................................... 19

5.4 Database ............................................................................................................................... 20

6. Operationalising the M&E Plan .................................................................................................. 21

6.1 Resources Required ............................................................................................................. 21

6.2 Revising the M&E Plan ......................................................................................................... 22

6.3 Ensuring Data Quality ........................................................................................................... 22

6.4 3-Year Schedule ................................................................................................................... 22

Annex 1: Evaluability Assessment ................................................................................................ 24 Annex 2: Three Possible Tools for the MAMPU Participatory Monitoring Toolbox ................. 37 Annex 3: Draft Methodology for Participatory Organisational Capacity Assessment ............. 38

Page 3: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan ii

List of Figures Figure 1: MAMPU Program Theory ....................................................................................... 4 Figure 2: Elements of Partner Project Performance Monitoring ............................................. 7 Figure 3: Steps to developing monitoring plan with partners ................................................. 9 Figure 4: KEQs over the life of MAMPU .............................................................................. 14 Figure 5: Scale for rating 'pointers' ...................................................................................... 41

Page 4: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan iii

ACRONYMS AND ABBREVIATIONS

ABCD Assets Based Community Development ARU Analytical and research unit CAPF Comprehensive aid policy framework EoPO End-of-Program Outcomes KEQ Key Evaluation Question M&E Monitoring and Evaluation POCA Participatory organization capacity assessment ROA RAPID Outcome Assessment SAC Strategic Advisory Committee SBA Strategic Based Approach

Page 5: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 1

1. INTRODUCTION

1.1 Why have an M&E Plan? This Monitoring and Evaluation (M&E) Plan (the Plan) describes a proposed M&E system for the MAMPU Program. This system addresses three interrelated purposes. Firstly, the M&E system supports partners, stakeholders, and policy audiences to learn from change. This includes the important contribution of robust and persuasive evidence on specific interventions to policy formulation and implementation. Secondly, this M&E system serves the accountability needs of key stakeholders, especially the partners, the MC, and AusAID, by documenting and demonstrating how MAMPU has contributed to its intended outcomes in the short, medium and long-term. Thirdly, and of no less importance, is the critical transformative role of M&E in MAMPU. MAMPU is explicitly designed to contribute to the empowerment of Indonesian women. This strongly implies a participatory orientation that provides the space for women to decide which change is important and to be actively involved in monitoring, reflecting on, and taking action on the basis of such change.

This document describes the conceptual outline of the M&E system for MAMPU. Further guidance on the operational aspects of M&E will be contained in two associated documents to be developed following approval of this plan:

• Toolbox of participatory monitoring methods; • Operational Guidelines for MAMPU M&E.

1.2 Principles underlying this M&E Plan The M&E system for MAMPU described in this document has been developed to be consistent with the following principles put forward in the program design:

Outcomes and process focused: M&E focuses on the quality of implementation processes carried out, and equally on what outcomes have occurred as a result of these activities. Outcomes can be positive or negative, expected or unexpected.

Gender equality: actively assess how the practical and strategic needs and opportunities of men and women have been affected - Monitoring and evaluation will examine how inequalities identified have been addressed and what impact they have had on the status of women and the freedom women have to meet their needs and those of their families.

Gives voice to those most marginalised: The M&E system will actively create opportunities for the perspective of the most marginalised (e.g. women, the elderly and people with disabilities) to be communicated directly to both internal and external decision makers. This has been addressed though the inclusion of participatory monitoring techniques.

Look for the unexpected as well as expected outcomes: Changes in the partners' design and implementation of its activities are expected as they learn about what works and does not work, and as changes occur in the context. The M&E should be sufficiently flexible to adapt to these changes.

Strengthens partners’ M&E systems: Specific approaches have been put forward to support partners to use and strengthen their own M&E systems.

In addition, the following three principles specifically guide the approach of the MAMPU MC team to M&E:

A user-focused orientation: The M&E system has been structured to provide the right information in the right format to key stakeholders when they need it.

Page 6: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 2

Learning-by-doing: Consistent with the approach to capacity development described in the MAMPU Capacity Development Strategy, this M&E Plan takes a practical ‘learning-by-doing’ approach to supporting partners.

A strengths-based approach: that builds on what the partners already do. Such an approach begins from the partners’ existing systems and processes.

1.3 Who is this M&E Plan for? The M&E system described in this document aims to address the main information needs of the key stakeholders involved in MAMPU. These stakeholders are the ‘audiences’ of the M&E System. Each audience requires particular information that reflects the different roles that they play in MAMPU management and governance.

The MAMPU MC is one such group. The Program design document describes the role of the MC as primarily supportive. The MC will assist the partners to carry out their projects while providing a range of capacity development support. This also encompasses program management functions including grant disbursement, and the provision of responsive flexible technical support services. The MC is expected to provide information on whole-of-program performance to other key stakeholders to enable them to play their roles in the program governance. To do this effectively the MC will need to collect, synthesize, analyse, and report a range of data on implementation of the MC’s own workplan, partner workplans, as well as key results including changes in partner capacity.

The Partners Forum plays a key role in the MAMPU governance structure. Comprised of representatives from partner organisations, AusAID, and Bappenas, the Partners Forum meets on a 6-monthly basis and is charged with setting and reviewing the direction of the MAMPU program. This includes reviewing the performance of the MC, approving priorities for capacity development initiatives directed at multiple partners, identifying priorities for research, and monitoring program risks. To play this role effectively, the Partners Forum will need synthesized information on whole-of-program implementation progress and outcomes, including the responsiveness and support provided by the MC. Progress in capacity development of each partner will also be particularly important. Risk monitoring will require information on the factors adversely affecting implementation progress (delivery risks), as well as the issues that influence and constrain progress towards MAMPU’s intended outcomes (development risks).

The Strategic Advisory Committee (SAC) will also refer to the information generated by the M&E system. Although the role of the SAC is still evolving, its anticipated function will be to ensure cutting edge global expertise and experience is applied to MAMPU. To make this advice strategic and relevant to MAMPU, the SAC requires information on the implementation progress of MAMPU partners, key results, and contextual factors that are impeding progress (risks).

AusAID and Bappenas are also audiences for the M&E system described in this document. They require information that can account for the resources being directed towards MAMPU. This includes assurance that sound financial and program management systems are in place, and that outputs are being delivered in line with the purpose of the program. More specifically, AusAID needs to know:

• Whether MAMPU is making satisfactory progress towards key outcomes; • The quality and reach of key outputs of the initiative; and • If the initiative is making adequate implementation progress against the annual plan

and budget.

Apart from the information requirements of the key groups described above, the information generated from the M&E system will also need to reach wider audiences that MAMPU seeks to influence. This wider group includes the parliamentarians, officials in relevant government

Page 7: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 3

agencies at the sub district, district, province, and national levels, as well as other civil society organisations active in the five MAMPU thematic areas and in womens empowerment more broadly. More detail on the processes to communicate this information to broader audiences is contained in the MAMPU Communications and Knowledge Management Strategy.

1.4 Organisation of this document This M&E Plan is organized as follows. Chapter 2 sets out in brief the key features of the ‘evaluand’ – the program that the M&E system aims to monitor and evaluate. Chapter 3 distinguishes ‘monitoring’ from the concept of ‘evaluation’ and describes a recommended strategy for monitoring MAMPU at two levels: the partner projects; and the overall program. Chapter 4 defines key evaluation-related terms and sets out the different types of evaluation that will be employed over the life of MAMPU. Chapter 4 also elaborates the role of the research in MAMPU and describes how this the program’s research agenda will be linked to M&E. In Chapter 5, the approach to managing information and the key reporting products is described. Finally, in Chapter 6, the resources required to implement the M&E Plan are explained, and an overarching workplan for the three years of this first phase is mapped out.

Page 8: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 4

2. WHAT DO WE WANT TO MONITOR AND EVALUATE? This section outlines the design of MAMPU with particular attention to intended results, summarises the findings of the Evaluability Assessment, and sets out the main implications for M&E.

2.1 Program Theory The design for MAMPU is underpinned by a simple idea: that collective, evidence-based advocacy, by strong gender-interested organisations, can catalyse widespread positive change in the lives of poor women across Indonesia by influencing key government policies.

The ‘program theory’ breaks this idea down into three broad stages of change: short-term, medium-term; and long-term outcomes. In the long-term (defined as 5-8) years, MAMPU hopes to see improvements in women’s access to better services and positive changes in their livelihoods. For these changes to occur, 3-5 years from now gender-interested organisations should be influencing targeted government policies and mobilizing greater public support for their cause. However, for this to happen, within 3 years, the capacity of MAMPU’s partners will need to be stronger, and they will need to be working better collectively. This process of change is represented diagrammatically in Figure 1 below. Figure 1: MAMPU Program Theory

To foster this change process, MAMPU supports selected partners to undertake projects in five thematic areas: (i) improving women’s access to government social protection programs; (ii) increasing women’s access to jobs and removing workplace discrimination; (iii) improving conditions for women’s overseas labour migration; (iv) strengthening women’s leadership for better maternal and reproductive health; and (v) strengthening women’s leadership to reduce

Page 9: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 5

violence against women. Alongside these projects, the MAMPU program will develop partners’ collective and individual capacity through a wide variety of learning and capacity developing activities.

However, this long-term process will not be linear and will be subject to influence from a range of contextual factors as the social, political, and economic environment evolves. Progress may seem painfully slow, only to leap ahead unexpectedly as unique windows of opportunity open up. As a result, the change process is likely to be highly unpredictable.

2.2 Evaluability Assessment An Evaluability Assessment (EA) was undertaken in June 2013, shortly after the commencement of the MC. The EA process reviewed the program theory and determined that there was sufficient basis for the program to be evaluable over its life. In particular, the EA determined that there is a sufficiently shared understanding of the outcomes of the program among key stakeholders. The complete EA is attached in Annex 1.

The short-, medium-, and long-term outcomes of MAMPU remain broadly defined at this early stage. This is appropriate for a program that is founded on a relationship of trust between partners and AusAID. It can be counterproductive for a donor to be too specific about outcomes at the outset of a program that intends to develop trusting relationships with civil society.1 Instead, it is imperative for this shared understanding to develop through an iterative learning process. However, as the AusAID M&E Standards emphasize, effective monitoring and evaluation of progress requires clarity around the End-of-Program Outcomes (EoPOs). For example, while improvements in organizational and collective capacity is the clear EoPO for phase I of the program, exactly what this looks like for each organization, network, and coalition is still to be articulated.

The experience of other AusAID programs that work with civil society highlights other lessons for M&E. As the same guidance document notes: “Reports suggest that conventional M&E based on simple indicator based assessment of predetermined outcomes is often inadequate to capture the wider and more complex results of such programs.”2 These concerns need to be balanced with the need for M&E to meet the accountability requirements of key stakeholders.

1 Kelly, L., David, R., & Roche, C. (2008). Guidance on M&E for civil society programs, AusAID: Canberra. 2 Ibid, page 26.

Page 10: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 6

3. MONITORING STRATEGY This section outlines processes for monitoring MAMPU performance at two levels: projects implemented by MAMPU’s partners; and the overall program. It explains how the MC will work with partners to support them to monitor their projects, sets out information requirements that apply across all partners, and describes how these will be brought together for overall program monitoring.

3.1 What is ‘Monitoring’? This plan acknowledges that monitoring is distinct from, but closely related to, evaluation. For the purposes of MAMPU, monitoring is defined as:

a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds (OECD-DAC, 2002).

Monitoring typically engages stakeholders in the regular assessment of the efforts of a project: inputs to the project (e.g. human and financial resources), the execution of key project activities within time and budgetary constraints, and the delivery of key outputs. Good practice increasingly emphasizes the need to regularly monitor the effects (or change) of all this effort.

The distinction between evaluation and monitoring can sometimes be blurred since both involve some form of data collection, analysis, and reflection on the implications for action. However, monitoring tends to be a continuous process, while evaluation is typically periodic and involves a greater element of analysis and reflection. Monitoring is integral for insightful evaluation because it provides a sufficient base of information is available about how a project or program was implemented, including whether and in what ways it deviated from its intended design. The approach to evaluation is described in Chapter 4 of this document.

3.2 Partner Project Monitoring MAMPU partners will be primarily responsible for monitoring the performance of their projects. At this level, MAMPU will apply a set of core monitoring questions, rather than common indicators. A question-based approach has several advantages in the context of a flexible partner-driven program with a diverse set of CSOs and projects. In particular:

• A question-based approach allows space to develop more detailed monitoring arrangements and indicators as partners progressively define their workplans during the first year of implementation.

• Core monitoring questions define consistent information requirements across all partners while enabling flexibility for each partner to determine how they answer these questions.

• A question-based approach supports a more focused approach to indicators and information requirements. The utility of indicators can be checked by reviewing them against agreed monitoring questions. Indicators that do not help to address the question can be discarded.

• Core monitoring questions support a more intuitive approach to analysis and action by helping to link information and indicators to its intended purpose. Understanding what indicators mean is clearer when data and information are collected and brought together to answer a question.

There is considerable diversity among partner projects. Nevertheless a number of basic elements are common to all. They are all based around an annual work planning cycle that specifies particular activities, and a corresponding budget. Each project describes, with

Page 11: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 7

varying degrees of clarity, processes to engage with relevant groups of women, men, policy-makers, parliamentarians and others to affect change (outcomes). Although presented in different formats in various partner proposals, these basic elements can be represented in a simplified generic logic model.

Figure 2: Elements of Partner Project Performance Monitoring

Five ‘core monitoring questions’ for MAMPU address key aspects of this simplified program logic. Key audiences need timely information on activity progress, implementation risks, financial expenditure, key outputs, and the ‘reach’ of these outputs among the groups that each project seeks to benefit or influence. Furthermore, the audiences of this M&E system need to know how these groups are responding to the outputs of partner projects and whether this is consistent with what was intended (effectiveness). The core monitoring questions that will apply across all MAMPU partner projects address these information needs. These core monitoring questions in turn form the basis of 3-monthly reporting by partners to the MAMPU program.

Core Monitoring Question 1 (Implementation Performance): Did we do what we expected to do?

The first monitoring question tracks implementation progress and is designed to build on the existing annual work planning cycle. Answering this question involves a simple comparison of planned quarterly activity with actual execution of activities.

Core Monitoring Question 2 (Financial Performance): Are the costs in line with what we expected?

The second question addresses financial management and prompts partners to compare planned expenditure to actual expenditure and identify any divergence.

Core Monitoring Question 3 (Risk): What challenges have affected our work?

The third monitoring question incorporates an analysis of delivery risks into the monitoring process. Each quarter the monitoring process prompts partners to consider the factors behind any divergence noted in the previous two questions. Over time, they will be encouraged to look for recurring risks or new risks. This approach encourages a more analytical and reflective approach to operational aspects of project management.

While the first three monitoring questions address basic project operational concerns, core monitoring questions 4 and 5 focus partner attention on outcome-oriented issues.

Core Monitoring Question 4 (Reach): Are we reaching the right people in sufficient numbers?

Inputs Activities Outputs/Reach Outcomes

Core Monitoring Questions 1, 2 & 3

Core Monitoring Questions 4 & 5

Page 12: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 8

The fourth core monitoring question addresses the crucial issue of ‘reach’. To contribute to the intended outcomes, partners need to understand who is participating and to consider if this level is sufficient. For example, if an intended outcome is to increase the awareness of female migrants of their rights, then partners need to know if women are participating in workshops or awareness-raising activities, and if so, how many or what proportion of women in a given locale are taking part. Answering this monitoring question could also involve reporting on engaging a range of local actors identified by the partners, for example men or local leaders. Whether these are the ‘right people’ will depend on the intended outcomes of each project.

Core Monitoring Question 5 (Outcomes): What changes and benefits are being experienced by participants in project activities?

All partner projects ultimately aim to make a positive change in the access and livelihoods of Indonesian women through a variety of direct and indirect approaches. Although this is a long-term process of change, it is important for partners to obtain rapid feedback on the effects of project activities in the short-term. This will encourage adaptive management within partners. It is also important for the monitoring system to provide the space for the participants themselves to shape and influence activities through the monitoring process. The fifth core monitoring question defines a focus on this type of monitoring information. It is deliberately structured as an open question, which enables partners to see and report intended change, but also to discover and learn from unplanned change.

3.3 Supporting Partners to Monitor Performance A priority of the MC will be to support MAMPU partners to monitor and learn from project performance. A learning approach to monitoring will contribute to the development of partner organisational capacity. Support for MAMPU partners will as much as possible be based on existing capacity monitoring processes and systems, in line with a strengths-based approach.

As of July 2013, MAMPU’s partners have developed project designs, annual workplans, and draft monitoring and evaluation arrangements. These M&E arrangements are presented in a variety of formats. The MAMPU core team has conducted a rapid appraisal of these documents and has consulted with each partner to:

• Clarify the proposed goals, outcomes, activities, key target groups, and the expected cause-effect relationships between these elements (the program logic);

• Clarify the analytical foundations of the focus proposed in the workplan and advising on technical content; and

• Advise on the coherence and consistency of the documents.

This support aims to refine project designs and the annual workplans prior to the finalization of grant agreements. During these consultations all partners indicated their strong interest in receiving further support and advice on M&E. M&E support has to date focused on clarification of the program logic. This process has not involved close attention to proposed indicators and measurement tools. Instead, it was felt that the focus of the MAMPU MC team would be best directed at ensuring that draft designs are a sufficiently clear basis for M&E. Particular issues for continued focus include the need for outputs to be expressed as ‘tangible’ products and services and the need for greater clarity on the groups of focus. The MC will continue to provide this type of support until grant agreements are signed with each partner, anticipated to be by the end of September 2013. Designs finalized at that point will provide the basis for the development of a partner-specific monitoring plan.

During the first 6 months, the MC will work closely with each partner to develop a monitoring plan that addresses each of the core monitoring questions prescribed above. This process is not intended to supplant existing monitoring arrangements. Rather, it will build upon the

Page 13: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 9

things that partners already do to monitor and track their work but jointly explore areas where additional tools can be used.

A 6 step process that will be followed that combines the structure of logic models with inclusive strengths-based approaches and participatory forms of monitoring. Steps 1, 2, and 3 involve a strengths-based exercise to draw out existing practices, introduce the core monitoring questions, and review suggested indicators. Steps 4 and 5 involve introducing, trialling, and selecting participatory monitoring tools. The output of this process will be a monitoring framework based on a shared understanding of the outcomes expected from each project (a logic model) and core monitoring questions to be answered through routine processes. Figure 3: Steps to developing monitoring plan with partners

Strengths-based approaches Strengths-based approaches (SBA) concentrate on the inherent strengths of individuals, organisations, communities, groups, sectors or networks to achieve development or change and for working in partnerships (Rhodes and Dureau 2011). SBA’s identify what is working in relation to achieving objectives and how to continue to use these strengths or how to move in the direction of change. M&E tools under the SBA umbrella focus on increasing learning and improvement within the context of the program, instead of focusing on why and where a program or activity did not achieve its objectives.

‘When people find out what is working well, they tend to be motivated to do more while focusing on weaknesses tends to be de-motivating and disempowering. In every context there is always something that has worked well (on which to build) and also existing resources which can contribute to shared efforts to achieve shared objectives. The respective strengths of partners are valued where there is greater chance of developing an understanding of different perspectives.’ (Rhodes and Dureau 2011)

Step 1: Start from where partner organisations are at

Step 2: Introduce core monitoring questions

Step 3: Review outcomes and indicators

Step 4: Introducing and trialling participatory tools

Step 5: Selecting tools

Step 6: Train and support partners to apply tools, analyse, reflect

Monitoring Plan based on core

monitoring questions

Page 14: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 10

SBA is both a philosophy about how to engage constructively with people, as well as a set of tools for practical application. These tools include appreciative inquiry, the most commonly used, Assets Based Community Development (ABCD), Sustainable Livelihoods Approach; and strengths-based research or evaluation methods.

Indicators to monitor partner projects Indicators will be part of monitoring partner project performance, but will be addressed in the context of core monitoring questions. All draft partner project designs have proposed indicators or refer to the need to identify these to measure the achievement of outputs, outcomes, and goals.

The MAMPU MC will work with each partner during the initial 6-month implementation period to review these indicators as part of finalizing the monitoring plan. This process will focus on indicators in two areas of each project, corresponding to core monitoring questions 4 and 5:

• Reach: specifically the inclusion of the key groups of people that are being targeted; and

• Outcomes: specifically the type of changes anticipated in the outcomes and goals in project designs.

A limited number of common indicators will be identified relating to two areas: AusAID’s Comprehensive Aid Policy Framework (CAPF); and key cross cutting concerns including gender and environment.

It is important to note that indicators may not be appropriate if there is considerable uncertainty over the intended outcomes of a particular partner project. In this context specific approaches to monitoring (such as the Most Significant Change methodology) may be more appropriate.

A toolbox of participatory and gender sensitive monitoring methods It is critical that the monitoring process enables the expected beneficiaries of MAMPU to actively participate, particularly poor and marginalized women. In this way M&E will be a transformative process that can itself contribute to the empowerment of its intended beneficiaries. Core monitoring question 5 is deliberately structured to create this space. Partners will address this question in their quarterly reporting.

Several partner proposals describe plans to apply a participatory approach to monitoring and evaluation, building upon existing experience and expertise that already exists. The MAMPU M&E system will actively encourage and augment this approach. To do this, a ‘toolbox’ of participatory monitoring methods will be developed by the MC during the initial 3 month period of MC implementation. These tools will be identified through a desk-based exercise that considers the unique requirements of MAMPU partner projects and reviews the strengths and weaknesses of a wide variety of tried and tested methods. Annex 2 sets out three participatory tools that have been proven in other contexts for consideration in the toolbox. This list is not exhaustive.

Once the toolbox is completed (see chapter 6 on operationalization of the M&E Plan), the tools will be introduced to partners and their sub-partners through a series of interactive workshops (see steps 4 and 5). This training will engage selected members of each partner in reviewing the project outcomes, the core monitoring questions, and consider the value of the existing tools and processes. Each tool will be introduced to participants through a learning-by-doing approach. At the conclusion, a facilitated session will engage participants in the selection of the participatory monitoring methods most suited to their projects. This approach places partners in the driver’s seat and increases commitment to using the methods. The MC, through the M&E Specialist and Partner Facilitators, will support the partners to apply participatory methods to monitoring and apply. The information collected through these tools will be collated against core monitoring questions 4 and 5 on a quarterly basis.

Page 15: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 11

3.4 Program level Monitoring At the program level, monitoring will involve data collection analysis in two areas: (i) MC implementation, including the quantity and quality of capacity development activities implemented with each partner; and (ii) joint learning from the implementation of partner projects.

Monitoring MC Delivery The MC will provide a range of support services to partners from grant administration to technical advice. This support needs to be flexible and timely and should be provided through processes that emphasize a collaborative and capacity development approach.

The development of systems and processes for managing the MC’s support is still ongoing. These systems will, when complete, cover an array of core MC functions including:

• Identifying, designing, and providing capacity development support activities; • Responding to requests for assistance from partners; • Procedures for engaging short-term advisory expertise; • Commissioning studies or analytical work; and • Financial and grant administration.

The efficiency and timeliness of these operational and administrative functions will be routinely monitored by the MC. This will be done by systematically recording key aspects of the process in a Management Information System3 and reporting aggregated information against simple metrics such as the time taken to respond to a request for technical expertise. These will be aggregated and used to inform internal monitoring of MC delivery.4

Joint Learning from Partner Project Monitoring In keeping with the partner-driven character of MAMPU, it is recommended that partners collectively play a key role in analyzing and making meaning from monitoring data. The MC will support this by analysing individual partner reporting to draw out common themes. This analysis will draw on the quarterly reports submitted by each partner and will distil the following information:

• Common factors influencing project implementation as reported by partners in Core Monitoring Questions, 1,2 and 3;

• The ‘reach’ of partner projects in each theme (from Core Monitoring Question 4); • Emerging feedback from target groups in each theme (from Core Monitoring

Question 5).

An annual structured learning process involving MAMPU partners would be facilitated by the M&E Specialist, Partner Facilitators, and other members of the MC team. This process would:

• Introduce synthesized monitoring information on factors affecting progress, reach, and outcomes from all partner projects;

• Use a participatory process to first rank, then discuss these factors; • Use small group exercises to identify strategies for common action in the coming 12

months.

The result of this process, will feed into the consideration by the Partners Forum of MAMPU’s research agenda, discussed in chapter 4 below.

3 For example, key information to record in the database will include the date that a request is received, the date that the support commences, date of completion. 4 AusAID will develop a separate contractor performance mechanism. However, collated data on MC responsiveness could be used to inform the contractor performance assessment.

Page 16: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 12

This approach ensures that partners occupy a central position in analyzing and determining action on the basis of monitoring data. While responsibility for this type of analysis is often allocated solely to the MC in conventional programs, this participatory approach better reflects the partner-driven nature of the MAMPU. The MC’s role in this learning process – to facilitate the process and document the results – also reflects its supportive function in the overall governance structure of MAMPU.

Page 17: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 13

4. A STRATEGY TO EVALUATE RESULTS This section presents the proposed approach to evaluation on MAMPU. It outlines several different types of evaluative exercise that will be applied during MAMPU’s life and describes how the key evaluation questions will be answered.

4.1 What is ‘Evaluation’? Following the definition of the OECD DAC, evaluation is:

The systematic and objective assessment of an ongoing or completed project, programme or policy, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors. (OECD DAC, 2002)

Also relevant, given MAMPU’s focus on empowerment and participatory approaches, is the definition of participatory evaluation:

Evaluation method in which representatives of agencies and stakeholders (including beneficiaries) work together in designing, carrying out and interpreting an evaluation. (OECD DAC, 2002)

Evaluation is an opportunity to reflect upon the approaches that worked well and those that did not work as well, to identify the reasons for success or failure, and to learn from both. Monitoring and evaluation are closely connected. Evaluation involves collating the information and data collected through ongoing monitoring processes, gathering additional primary and secondary data and information to draw conclusions about effectiveness, sustainability, impact (if possible), sustainability, and relevance. Evaluation is difficult in the absence of a sufficiently comprehensive base of monitoring data and information on program implementation. At the same time, evaluation can also unearth interesting, hitherto unexpected results that stakeholders may wish to incorporate into future monitoring.

4.2 Key Evaluation Questions (KEQs) MAMPU’s design identifies four KEQs that define the focus of evaluative activity over the life of the program5:

1. How and to what extent has the program affected the partners and networks’ capacity to influence government reform?

2. How and to what extent have the partners and networks influenced government reform in relation to the needs and priorities for poor women?

3. How and to what extent has the program contributed to improved access and livelihoods for poor women?

4. What is changing in the context, and how should the program respond?

KEQs 1, 2, and 3 align with the short-, medium-, and long-term outcomes intended by the program, while KEQ 4 will be answered through routine MC monitoring. KEQ 1 will be the main focus of MAMPU’s M&E system during the first 3 years of implementation. However, later in the program life evaluative work will need to address KEQ 2, and eventually KEQ 3. Each KEQ will progressively build a base of information so that by the final 2 years of the program, a comprehensive statement can be made of MAMPU’s contribution against to the three levels of the theory of change.

5 Minor amendments have been made to these to improve their suitability as evaluation questions.

Page 18: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 14

Figure 4: KEQs over the life of MAMPU

Monitoring data on partner projects (collected through against the five core monitoring questions described in the previous chapter) and MC capacity development activities will provide an essential base of information for all three KEQs. This type of data will be used to capture the quantity and quality of program effort and help to assess the case for MAMPU contribution to outcomes at each of the three levels. Information on the reach of key outputs of partner projects as well as the experiences of the key groups reached (core monitoring questions 4 and 5) will be particularly important for this purpose.

4.3 KEQ 1: How and to what extent has the program affected the partners and networks’ capacity to influence government reform?

MAMPU will apply two types of evaluative process to address the first KEQ: (i) annual Participatory Organisational Capacity Assessments; and (ii) three network mapping studies (baseline, mid-term, and endline). This will ensure that KEQ 1 can be comprehensively addressed by year 3 of implementation and at key points in the program life.

Participatory Organisational Capacity Assessments The MAMPU MC will facilitate an annual participatory assessment of partner organisational capacity. A simple workshop-based capacity assessment tool based on the 5Cs approach to capacity (see Box 1 below) will be applied. The methodology involves working with participants from each organization to develop a capacity assessment framework based on four capabilities (simplified versions of the 5 core capabilities listed above). Once developed, the framework is applied by participants to score different aspects of their organisation and discuss the results. The final part of the workshop moves from assessment to identifying and ranking priorities for the coming 12 months. While the tool generates numbers through the scoring and ranking process, its real value lies in the in depth qualitative discussion around capacity issues. During the initial round, participants will be asked to refine how organization should look in 3 years, and how this translates to each of the four core capabilities. A draft methodology for the POCA is described in Annex 3. This draft will be modified further by the M&E Specialist and CD Coordinator prior to being applied with partners. This process will ensure that the tool sufficiently addresses areas of particular interest to MAMPU (such as advocacy). Other areas of modification will assess the need to incorporate external perspectives through, for instance, key informant interviews of key clients and donors.

Years 2-3: KEQ 1

Short-term Outcome Improved capacity and readiness among partners and networks

Years 3-5: KEQ 2

Medium-Term Outcome Positive changes in voice and influence on policy

Years 5-8: KEQ 3

Long-Term Outcome Positive changes in access to services and livelihoods

Page 19: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 15

The number and composition of participants in the POCA will be discussed with each partner organisation but should reflect the need for a selection of key/comparable roles within the organisations. This will minimise the issue of individual people leaving/changing over time.

Each annual assessment will be facilitated by Partner Facilitator working closely with the M&E Specialist and the CD Coordinator. The results of the process will feed into the development of each partner’s annual workplan and CD plan.

Through successive annual rounds, the POCA process will document partner perceptions of changes in their capacity, and progress towards the vision of capacity outlined in the initial round.

Organisational network mapping studies The MC M&E Specialist will design and commission three organisational network studies over the life of MAMPU to assess changes in the capacity for collective action. These will be implemented at three points: baseline (within the first 12 months); mid-term (2-3 years); and endline (5-7 years).

These studies will complement the annual POCA process to ensure KEQ 1 can be comprehensively addressed by years 3 and 7 of the program. These studies will be designed to be tightly focused on a limited number of indicators within the following three broadly overlapping categories6:

• Network density: including participation, leadership, sustainability, and resilience; • Network vibrancy: including the nature of the relationships within the network, and

the degree of connectedness and centrality; • Network outcomes: the policy being targeted, and major changes in that policy.

The studies will focus on describing the network of organisations (at the national and local levels) that relate to a particular policy in each of MAMPU’s five themes. This is important to ensure that the studies are sufficiently concentrated in areas of intended partner influence to show change over the life of the program. A complete mapping of the organisational network

6 See Scaling Impact and Keystone Accountability (June 2010), “Next Generation Network Evaluation Innovations” http://www.scalingimpact.net/files/IDRC_Network_IPARL_Paper_Final_0.pdf accessed 24 July 2013

Box 1: MAMPU’s approach to capacity development. The MAMPU Capacity Development Strategy defines capacity as “that emergent combination of individual competencies, collective capabilities, assets and relationships that enables a human system to create value” (ECDPM, 2008). Consistent with this, capacity development is: “the process of enhancing, improving and unleashing capacity; it is a form of change which focuses on improvements” (ECDPM, 2008).

The MAMPU Capacity Development Strategy builds on the five capabilities model of capacity development (the ‘5Cs model’) that emphasizes endogenous approaches to capacity change. The 5 core capabilities are:

1. the capability to survive and act; 2. the capability to generate development

results; 3. the capability to relate; 4. the capability to adapt and self-renew; 5. the capability to achieve and maintain

coherence.

Changes in capacity, according to this approach, emerge from the interaction of these capabilities. Importantly, these include the capability to develop and maintain networks and relationships with other organisations, an area where MAMPU intends to focus. A holistic process of capacity development is primarily internally driven and can only be supported and facilitated by external entities such as MAMPU. This approach also implies that evaluating changes in capacity should be participatory and inclusive.

Page 20: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 16

in the broad thematic areas described in the program design across Indonesia, is likely to be an overly resource intensive exercise.

The organizational network mapping baseline study will be designed and implemented during the first 6 months of the 2014 calendar year. This will allow sufficient time for partners to articulate their policy focus and to collaborate with others to establish policies of common interest.7 This will also enable the studies to describe the particular policy being targeted by the network at the baseline stage, and enable changes in this policy to be identified at the mid-term and endline points.

The organizational network studies will employ a mixed method approach that uses qualitative methods to probe particular issues in more depth with key informants, describe key features of target policies, combined with a survey tool to collect quantitative data to establish wider patterns in the selected indicators. The methodology, including the associated tools and analytical procedures will be described in detail at the baseline stage so that the study can be repeated in year 3 and between years 5 and 7.

4.4 KEQ 2: How and to what extent have the partners and networks influenced government reform in relation to the needs and priorities for poor women?

While KEQ 1 addresses changes in capacity, KEQ 2 focuses on the extent to which it has been applied to influence the particular target policies. Positive changes in this medium-term outcome are expected to first emerge between years 3 and 5 of the program. Consequently, MAMPU must be in a position to put forward a comprehensive answer to KEQ 2 at this stage and again by year 7.

To do this, the MC will draw on information from two types of evaluative process: (i) the Organizational Network Mapping Study in years 1, 3, and 7; and (ii) at least 2 studies of policy change to be undertaken between years 3 and 5 and again in year 7.

The Organisational network mapping studies will document the networks in each thematic area and describe the particular characteristics of the policies being targeted. The repeat of these studies in years 3 and 7 will identify any key policy changes in areas where MAMPU networks have had a particular focus. However, these studies will not be positioned to explore and learn from instances of policy change in more depth. This deeper analysis will occur through specific studies of policy change.

The MAMPU MC will commission at least 2 in depth studies of policy change over the life of MAMPU using the RAPID Outcome Assessment (ROA) methodology developed by the Overseas Development Institute (ODI) in the UK. There will be at least 1 study in years 3-5, and 1 additional study in year 7.8 This tool, which draws heavily on the Outcome Mapping methodology, is a participatory and collaborative approach to exploring how an identified policy change came about and the particular factors that contributed.9 The ROA methodology involves 3 main stages. During the first stage, the team conducts background research around the policy change of interest. The second stage involves a workshop facilitated by the study team during which stakeholders define the policy at the start of the 7 This is presently unclear across most themes. For example, the project design submitted by Kapal Perempuan aims to influence social protection policy but plans to select the particular scheme of focus during the first 12 months of project implementation. 8 See ODI (2012), ‘RAPID Outcome Assessment’: Guidance Note, http://www.odi.org.uk/sites/odi.org.uk/files/odi-assets/publications-opinion-files/7815.pdf accessed 13 August 2013 9 Where no change has occurred in a targeted policy, these studies will instead explore the factors that have contributed to this situation. In either case (change or no change, as identified by the organizational network study), the particular policy for the ROA will need to endorsed by the Partners Forum, including AusAID and Bappenas.

Page 21: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 17

period and the end, identify key policy actors and corresponding changes in their behaviour. Essentially, this process involves building a ‘map of influence’ and often involves the use of visual methods such as timeline charts. The third and final stage involves follow up by the study team to refine the narrative and further develop stories of change from individual actors in the policy process. The product from this study will be a detailed and in depth exploration of how a specific instance of policy change has come about and the relative contribution of MAMPU partners and networks. This analysis will be a particularly valuable learning opportunity for MAMPU stakeholders, because it will set out a narrative of policy change within a rich understanding of context.

4.5 KEQ 3: How and to what extent has the program contributed to improved access and livelihood for poor women?

KEQ 3 addresses the ultimate goal of MAMPU – positive changes in the access to services and livelihoods of poor women in Indonesia. The program design anticipates these changes to emerge between years 5 and 7 of MAMPU implementation. Ensuring adequate information to establish change at this level will be challenging. The theory of change underpinning MAMPU holds that wider impacts on poor women will occur through the application of more gender-sensitive policy. The specific aspects of these policies that will be targeted have not been defined in most themes.

However, the MAMPU MC will support partners to collect data relevant on this KEQ in specific local contexts. Firstly, this will be done through the monitoring process which will incorporate participatory forms of monitoring, as outlined in chapter 4. Secondly, once partners have further refined their project designs over the first 12 months, they will be supported to undertake quantitative and qualitative baseline studies, where appropriate. These baselines will, where suitable, enable change in access and livelihoods of poor women to be assessed in very specific geographic locales where partner projects work.

As the focus of partner and network advocacy efforts is refined, opportunities will emerge to systematically collect data on aspects of women’s access to services and livelihoods relevant to policies that the program will target. The MC will continuously monitor this emerging focus with a view to identifying when and in what form additional data collection and analysis should be done.

4.6 Special Research Studies Research plays a special role in MAMPU. The plausibility of the theory of change rests partly on the credibility of the evidence that informs the advocacy of partners and networks. Recognising the unique demands of this agenda, the program has access to an Analytical and Research Unit (ARU) comprised of specialized researchers with a range of expertise. The ARU is critical to MAMPU’s evidence-based advocacy agenda.

The processes governing the management of the ARU and the identification of research priorities are presently still being developed by the MC in consultation with AusAID. Broadly, research studies undertaken by the ARU will fall into two categories: (i) formative research; and (ii) evaluative research.

‘Formative research’ will involve studies that are forward looking and more exploratory in nature. They will be identified and guide future efforts of the partners. They will be more broad-based and ‘exploratory’ and could be structured around exploring the state of play in a particular theme.

‘Evaluative research’ will be more focused around a specific research question. Studies of this kind would involve rigorous research on particular interventions (not necessarily entire

Page 22: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 18

projects, but more likely to be a very specific intervention within a partner project) to generate robust evidence for policy advocacy.10

Other monitoring and evaluation processes will link closely with the ARU’s analytical agenda. Routine monitoring will identify promising interventions that could potentially be the subject of more intensive research by the ARU. However, it is likely that these will need to be tested and investigated further before producing evidence that stands up to scrutiny.

The 6-monthly Partners Forum will be the key mechanism for considering and prioritizing ideas for MAMPU’s annual research program. These ideas will need to be, in the first instance, based on the advocacy interests of the partners. The process by which the ideas are reviewed, refined, clarified and either considered for a more detailed research plan or rejected is presently being developed. However, it is likely to involve a specific facilitated session during the Partners Forum during which potential ideas would be ranked by participants. Preparation for this session may involve ‘a long list’ of ideas in the ‘formative’ and ‘evaluative’ categories. The involvement of the partners through the Partners Forum is important to establish ownership and ensure research is aligned with their advocacy agenda. The product of this session will be a shortlist of ideas for the research. Once this is available a process of refinement with the ARU will be needed to ensure that each research is feasible.

4.7 Whole-of-Program Performance Analysis It is recommended that three Whole-of-Program Performance Analyses be developed over the life of the program, each drawing together the evidence available through M&E and research processes to set out the case for program contribution to each level of outcome.

The first Program Performance Story Analysis should be developed in year 3 of MAMPU, just prior to the conclusion of the first phase of MAMPU. Facilitated by the M&E Specialist, this will draw together all evidence from:

• Monitoring processes at partner and program level; • Successive POCA rounds with each partner; • A comparison of the baseline and mid-term organizational network study; • If completed, a ROA study of policy change; and • Relevant research completed by the ARU.

Information from these processes will be used to tell a program performance story consisting of three parts:

• Verified changes in targeted policies; • Changes in organizational and network capacity; • MAMPU activities at program and partner level that contributed to these changes.

In effect, this Whole-of-Program Performance Story process draws discrete M&E processes together to address KEQ 1. The results will feed directly into the Program Completion Report (see chapter 5). A similar process will be conducted at years 5 and 7 drawing on additional evidence available at that stage.

10 One example of this type of research is the ‘Trip Advisor’ concept that proposes to use SMS-based information to provide prospective female migrants with information on the quality of available migration agents. This study, which links closely to the Migrant Care project in Theme 2, aims to generate robust evidence of the use of the right information to positively influence behaviour.

Page 23: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 19

5. MANAGING AND REPORTING M&E INFORMATION This section describes how the information collected through the various monitoring and evaluation processes will be brought together for reporting to key audiences. This section also describes how data will be entered, stored, and managed in a program database.

5.1 Routine Reporting Products Over the life of MAMPU, the MC will communicate information from the MAMPU M&E processes to key audiences (Partner Forum, AusAID, Bappenas, SAC) through two reporting products: Six-monthly workplans; and Annual Reports.

Six-monthly workplans will be delivered in May and November of each calendar year. These workplans will include:

• Progress against the Annual Workplan and Budget at the partner and program levels. • The reach of partner projects and the emerging outcomes for the reporting period,

and the plan for delivery of products for the second six-month period; • Key challenges and the management decisions to respond to and manage future

risks so as to ensure implementation of activities and delivery of products at the expected quality.

The MAMPU Annual Plan will be submitted by July 31st each year and will be the principle formal vehicle for communicating more in depth analysis of progress and results from M&E processes. The Annual Plan will contain:

• An analysis of the political, economic and social context for MAMPU and a comment on the ongoing relevance of the program short, medium, and long-term outcomes;

• An analysis of progress towards short-, medium-, and long-term outcomes. This will include a summary of partner project progress (including highlights of participatory monitoring data), a synthesis of partner participatory organization capacity assessments, and analysis of the factors shaping and constraining progress;

• An analysis of the functionality of the M&E system and any proposed amendments and alterations;

• A description of progress against the MC’s workplan and budget, and the factors influencing this;

• A description of the workplan for the following 12 months.

5.2 Program Completion Report The Program Completion Report to be completed by the MC in year 3 will conform to AusAID requirements and will incorporate the results of the Whole-of-Program Performance Analysis described in chapter 4. This will provide a comprehensive picture of MAMPU’s performance against KEQ 1 which focuses on MAMPUs short-term outcome.

5.3 Key meetings Meetings will be an important vehicle for communicating the results of M&E processes. The key points from six-monthly workplans and Annual Plans will be presented by the MC to the Partners Forum for consideration and endorsement. In addition to formal reporting mechanisms, M&E information will be communicated through the Partners Forum through participatory processes. As suggested in chapter 3, synthesized monitoring information from all partner projects will be presented at the forum to enable a collaborative approach to analyzing monitoring data.

Page 24: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 20

5.4 Database A user-friendly database will be established to support the efficient management of information from the various M&E processes described in this document:

• Grant management-related financial data including progress against budget and activity acquittals;

• Reporting status against due date; • Implementation progress, reach and outcome data from partner projects; • Reported risks affecting partner implementation progress (core monitoring question

3); • Capacity development and learning activities facilitated through the MC, including

key information on efficiency and responsiveness including the dates of requests for assistance, activity designs, procurement, and mobilization.

A full-time Database Officer will be recruited to design the system and ensure that it captures required data. Upon completion the system should be able to generate simple reports basic status information on each partner project, as well as the status of all MC capacity and learning activities.

Page 25: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 21

6. OPERATIONALISING THE M&E PLAN This section described how the M&E processes described in this M&E Plan will be operationalised. It identifies the key resources and expertise required, and summarises how they will be deployed over the initial 3 years of the program life.

6.1 Resources Required A range of expertise will be required to implement MAMPU’s M&E system over the first 3-year phase of the program. Table 1 below outlines the key elements of the M&E system proposed of MAMPU and identifies the key resource requirements. The 3-year overarching schedule over the page provides the indicative timeframe for the engagement of resources and expertise additional to the M&E Specialist. Approximately 10 per cent of MAMPU’s budget has been allocated to M&E in recognition of the important role of evidence in the program. A more detailed budget will be developed subsequent to the submission of the draft M&E Plan. Table 1: Resources required for MAMPU M&E Plan

Aspect of M&E System Resources required

Support to Partner Project Monitoring (ongoing)

M&E Specialist, M&E Officer, 3 x Partner Facilitators

Overall Program Monitoring and Reporting (ongoing)

M&E Specialist, M&E Officer, Database Officer

Toolbox of Participatory Monitoring Methods and Training (to be decided)

National or International Consultant with expertise in gender-sensitive participatory monitoring methods and ability to design and carry out associated training

Participatory Organisational Capacity Assessments (Annual)

Partner Facilitator, CD Coordinator, M&E Specialist

Organisational Network Mapping Studies (x2)

Research institution and individuals with capacity in qualitative and quantitative methods of social network analysis and familiarity with Indonesian policy processes

Policy Change Studies (at least 2) Research institution and individuals with in depth knowledge of Indonesian policy environment and expertise in qualitative and participatory research methods

Special research studies (To be determined)

Analytical and Research Unit incorporating high level skills in experimental, quasi-experimental, and qualitative research design.

Overall responsibility for operationalizing of the M&E Plan and ongoing implementation will rest with M&E Specialist (reporting to the Team Leader) in the MAMPU MC Core Team. The engagement of an additional M&E Officer and a Database Officer who will work with the M&E Specialist will form a MAMPU M&E Unit who will be dedicated to operationalizing,

Page 26: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 22

maintaining, and revising the system. Support for partners will also draw heavily on the CD Coordinator and the Partner Facilitators who will act as coordinators for all MAMPU capacity development support to each partner.

6.2 Revising the M&E Plan Each year the M&E Specialist will consult with program stakeholders to assess the utility of the system. Questions that will guide this assessment will include:

• Is data collected being used to its maximum potential? • If not, was this because the framework does not target the right areas? • Have stakeholder requirements changed over the life of the program?

6.3 Ensuring Data Quality The monitoring system will depend to a great extent on the quality of data collected by partners against core monitoring questions. The M&E Specialist and M&E Officer will develop and apply a simple procedure to ensure the quality of monitoring data, particularly against core monitoring questions 4 and 5. Since ‘quality’ in participatory data is closely related to the process of facilitating the use of a tool, this will involve observing the way tools are applied in a range of locations. Other methods and practices that will be used to encourage data quality include:

• Regular refresher training for field staff in the participatory monitoring tools adopted; • Feeding back data collected through participatory methods to participants as a

routine practice; • Encouraging where possible partners and sub-partners to use local staff who are

familiar with the locale where the participatory tool is being applied.

6.4 3-Year Schedule An overarching 3-year schedule for implementation of the MAMPU M&E system is depicted below. This provides an indicative timeline for key monitoring and evaluation activities and reporting events. It should be noted that the timescales will be dependent on the timeframe for approval of the M&E Plan.

Page 27: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 23

ACTIVITY 2013-2014 2014-2015 2015-2016J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J

MONITORING

Strengthen Partner Project Designs X X X

Develop Participatory Monitoring Toolbox X X

Develop Operational Guidelines for Monitoring X X

Develop Partner Monitoring Plans X X X X

Training in Participatory Monitoring Tools X X

Support and provide additional training to partners on monitoring tools

X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X

Recruit M&E Officer X X

Recruit Database Officer X X

Database development X X X

REPORTING

Partner Quarterly Reports X X X X X X X X X X X

MC Six-Monthly Workplans X X X X X X

MC Annual Reports X X

Partners Forum/Thematic Working Group X X

EVALUATION

Participatory Organisational Capacity Assessments X X X X X X X X X

Organisational Network Mapping Baseline Study X X X

Organisational Network Mapping Mid-Term Study X X X

Policy Change Study using ROA methodology X X

Whole-of-Program Performance Analysis X

Program Completion Report X

Independent Completion Report X

RESEARCH

Develop processes for identifying annual research priorities X X

Engage ARU personnel X X

Identify research priorities X

MAMPU M&E Plan 3-Year Schedule

Page 28: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 24

ANNEX 1: EVALUABILITY ASSESSMENT

EVALUABILITY ASSESSMENT

MAMPU PROGRAM

JUNE 2013

Page 29: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 25

Acronyms and Abbreviations ARU Analytical and Research Unit

AusAID Australian International Development Assistance

BAPPENAS State Ministry of National Development Planning or Badan Perencanaan dan Pembangunan Nasional

C1 and C2 Component 1 and Component 2

EA Evaluability Assessment

GOA Government of Australia

GOI Government of Indonesia

ILO International Labour Orgnaisation

Kapal Perempuan Lingkaran Pendidikan Alternatif (Circle of Women’s Alternative Education)

Komnas Perempuan Komisi Nasional Anti Kekerasan Terhadap Perempuan (National Commission on Violence Against Women)

KPI Koalisi Perempuan Indonesia Untuk Keadilan dan Demokrasi (Indonesian Women’s Coalition for Justice and Democracy)

KPP-DPD Indonesian Women Regional Representatives’ Caucus

KPPI Indonesian Political Women’s Caucus

KPPRI Indonesian Women’s Parliamentary Caucus

MAMPU Maju Perempuan Indonesia untuk Penanggulangan Kemiskinan (Empowering Indonesian Women for Poverty Reduction)

MC Managing Contractor

Migrant Care Perhimpunan Indonesia untuk Buruh Migran Berdaulat ( Indonesian Association of Migrant Workers’ Sovereignty)

M&E Monitoring and Evaluation

NGO Non-Government Organisation

PEKKA Pemberdayaan Kepala Keluarga Perempuan (Women Headed Household Empowerment)

Yayasan BaKTI Yayasan BaKTI – Bursa Pengetahuan Kawasan Timur Indonesia (Eastern Indonesian Knowledge Exchange Foundation)

Page 30: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 26

1.0 Introduction

This MAMPU Program Inception Phase Evaluability Assessment (EA) considers whether the Project will be able to be assessed during implementation and immediately afterward, upon completion. It focuses on:

1. Shared Stakeholder Understanding

• Assessing whether key stakeholders have a shared understanding of the Project

2. Overall Project Design

• Establishing how well MAMPU goals and objectives, components, themes and associated programs have been identified, defined, and integrated

• Reviewing the program logic and theory of change with key stakeholders and describing the extent to which the program can be evaluated including clarity of expression of end-of-program outcomes

3. Working with Partners

• M&E capacity needs - assessing the capacity of the implementation team and partners to participate in the design and/or conduct of M&E activities

• Identifying program working relationships with partners to ensure mutual accountability

• Identifying key evaluation questions and information of interest to stakeholders

4. Key M&E Issues

• Identification of reporting requirements for users of the M&E system-generated information, including program partners, AusAID, GoI and the Managing Contractor (MC)

• Reporting requirements for users of the M&E system-generated information – i.e. progress reporting, Quality at Implementation Reporting, Annual Program Performance Reporting, and how the M&E plan will provide evidence for reporting against the Performance Assessment Framework.

• Examining potential data sources to ensure quality data is collected and analysed as expected within the required reporting cycles

• Reviewing cross-cutting policy areas that need inclusion in the M&E Plan

• Identifying issues and/or constraints that will affect the design of the M&E Plan.

• Reviewing budget and resources available for M&E activities

2.0 Shared Understanding

The design of the MAMPU Project is the result of a highly participatory, multi-stakeholder consultative process that took place between July 2011 and March 2012. Led by AusAID, it included discussions with representatives of several Indonesian Central Government Ministries, some of their Australian counterparts, as well as workshops with central and district level representatives from a number of key Indonesian women’s and gender-interested NGOS (non-profits). A project design expert, several gender experts, private sector representatives, and other stakeholder groups also contributed to MAMPU’s design. During this wide-ranging consultative process it was agreed that (i) strategic oversight and management of MAMPU would be given to the State Ministry of National Development

Page 31: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 27

The word service is implicit in the Goal statement. It would be clearer to all stakeholders if this was explicitly stated as ‘access to services’.

Planning11 (BAPPENAS - Badan Perencanaan dan Membanguan Nasional) and that (ii) the main objectives and focus of the Project are:

• Project Goal - Improved access and livelihoods for poor women in Indonesia in selected geographic areas within the targeted themes

• Project Purpose - To build networks and coalitions led by strengthened women's and gender interested organisations who will influence government policies, regulations and services, and in selected private sector arenas, in order to improve the access of poor women to critical services.

• Five Themes (GoI and the Australian Government priority areas for reducing poverty among women in Indonesia) will guide MAMPU interventions. They are:

(i) improving women’s access to government social protection programs

(ii) increasing women’s access to jobs and removing workplace discrimination

(iii) improving conditions for women’s overseas labour migration

(iv) strengthening women’s leadership for better maternal and reproductive health

(v) strengthening women’s leadership to reduce violence against women

• Project Components –Component 1 – Analysis, action, and advocacy by networks led by national and local women’s and gender-interested organisations, and Component 2 - Women in Parliament, supporting caucus’s women parliamentarians and male gender advocate parliamentarians (national/district levels) to engage with women’s organsiations to advance gender equality reforms within the targeted themes.

This common and well-researched understanding of the key building blocks of MAMPU (Figure 1) increases the likelihood of the Project’s evaluability during implementation and at Project end.

3.0 Clarity of Objectives, Logic and Theory of Change

3.1 Clarity of Objectives

3.1.1 Goal and Purpose The MAMPU Goal is straightforward and simply stated - Improved access and livelihoods for poor women in Indonesia in selected geographic areas within the targeted themes. Further, it is endorsed by all key stakeholders, including the host and donor governments, and is expressed in terms of expected impacts that can be measured.

MAMPU’s Purpose has three parts: (i) To build networks and coalitions led by strengthened women's and gender interested organisations (ii) who (sic - that) will influence government policies, regulations and services, and in selected private sector arenas, (iii) in order to improve the access of poor women to critical services.

11 www.bappenas.go.id

Page 32: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 28

Progress and achievements of partner interventions related to this statement are measurable. Program design documents12 also provide definitions of its key phrases (‘networks and coalitions’, ‘strengthened organisations’, ‘changes to government policies, regulations, and services’ and ‘improving the access of poor women to critical services’).

The Purpose statement also clearly states that the task of building networks and coalitions is the responsibility of strengthened women’s and gender-interested organisations. In other words, partners that have already had some form of capacity development.

3.1.2 MAMPU Themes and Components The MAMPU Program’s five Themes (Figure 1) are simple and self-explanatory, and were selected after lengthy stakeholder consultations and research. These are the pillars that will guide Program interventions with partners, and their networks and coalitions. They are also the main areas where improvements to the lives of poor women will be expected at Program end.

The MAMPU Objectives Tree (Figure 1) is also made up of two core Components that are well-researched and were also developed after lengthy consultations with key stakeholders and experts. They have also had their key words and phrases defined in Program documents.

12 MAMPU Indonesia: Empowering Indonesian Women for Poverty Reduction, Program Design Document, Part A, September, 2012, AusAID.

(1) The term ‘improved livelihoods’ that is in the Program Goal, however, is not included in the Purpose statement nor in the two lower level Component statements. It is recommended that it is included in the Program Purpose to ensure it fits more closely with the Goal. (2) ‘Selected private sector arenas’ is part of the Purpose but is not defined nor is it addressed again in sections of Parts A and B of Project design documents.(3) Whether partners’ clients (i.e other community organisations or groups of direct beneficiaries) should also receive support from the MAMPU Program is not addressed in Program design documents. It is suggested that if partners’ capacity development initiatives can be strengthened by including their clients then they should be included in these exercises as this will help to achieve the Program’s goal. This is also the approach Program design documents recommend for the inclusion of women’s caucuses and gender-interested male parliamentarians in MAMPU.

Care nevertheless needs to be taken to ensure the C1 only relates to women’s organisations, gender-interested organisations, and potentially their client organisations and direct beneficiaries, while C2 is broader and includes caucuses of women parliamentarians and male gender advocate parliamentarians as well.

Page 33: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 29

(1) There is some potential for overlap between each of these sub-components – CI.1 refers to individual organisations, C1.2 to individual organisations and coalitions of organisations, and C1.3 networks of organisations. To ensure clear distinctions are made between these sub-components it is recommended that C1.1 refers to only individual organisations, C1.2 focuses on only support to coalitions (as defined in Program documents), while C1.3 focuses only on learning and networks as defined by Program documents. (2) Further, reflecting on the Program’s Goal and Purpose and C1 (as well as C2), it is evident that the private sector has been overlooked. It is suggested that this oversight is addressed too.

Figure 1: MAMPU Objectives Tree

3.1.3 Component 1 Program Design documents make it clear that Component 1 (C1) will be the prime focus of interventions with local partners in the short to medium term. For this reason it has been developed further in Program design documents (Figure 1):

C1.1 strengthening women’s and gender-interested organisations

C1.2 supporting analysis, action, and advocacy by individual and coalitions of women’s and gender- interested organisations, and

C1.3 networking and learning

Key terms associated with these immediate results have also been discussed in a participatory manner with key stakeholders and experts, and defined. They can be found in Program documents.

Goal: Improved access and livelihoods for poor women in Indonesia in selected geographic areas within the targeted themes (IMPACT)

Purpose (Strategic Objective 1): To build networks and coalitions led by strengthened women's and gender interested organisations who will influence government policies, regulations and services, and in selected private sector arenas, in order to improve the access of poor women to critical services. (LONG-TERM OUTCOME)

Theme 1: Access to Social

Protection & Poverty

Reduction P

Theme 2: Increasing

Women’s Access to Jobs and Removing W k l

Theme 3: Improving Conditions for Women’s Overseas L b

Theme 4: Strengthening Women’s Leadership for Better Maternal and Reproductive Health

Theme 5: Strengthening Women’s Leadership to Reduce Violence Against

Component 1 (Strategic Objective 1.1): Analysis, Action, and Advocacy by Networks led by National and Local Women’s and Gender –Interested Organisations

Component 2 (Strategic Objective 1.2): Women in Parliament, supporting individual women and male gender advocate parliamentarians and parliamentary caucuses (national/district levels) to engage with women’s organisations to advance gender equality reforms within the targeted themes.

1.1 Organisation Strengthening

1.2 Analysis, Action, Advocacy

Page 34: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 30

3.1.4 Component 2 Component 2 (C2) aims to support caucus’ of women parliamentarians and male gender advocate parliamentarians to engage with women’s organisations to advance gender equality reforms within MAMPU’s targeted Themes. While some C2 interventions are likely to begin in 2012 this Component will gain more prominence after the April, 2014 Indonesian elections. For this reason it has not been fully developed in Program design documents.

3.2 MAMPU Logic Model and Theory of Change Figure 2 overviews the MAMPU Program Logic Model. The rationale for it is grounded in a strong base of evidence that empowered women can make a real difference13.

The logic of cause and effect of the MAMPU Program is that:

a) Support to well-established, broad-based women’s and gender-interested organisations can make a difference to the lives of poor women

b) Support to stronger individual organisations to form networks and coalitions amongst themselves and with women’s caucuses, and women and male gender-advocate Parliamentarians, will make them even more powerful

c) With greater strength coalitions will be able to more effectively advocate and this will bring about policy, regulatory, and government services changes that will help poor women get access to services and improved livelihoods

Figure 2: MAMPU Logic Model

13 “When spider webs unite they can tie up a lion,” Ethiopian proverb.

In many ways C2 bears close parallels to C1. Both are based on the notion that strengthened networks and coalitions of organisations and individual organisations can help to improve the lives of poor women in Indonesia. For this reason C1 sub-component (ii) and (iii) may be applicable to C2. This issue should be addressed in consultation with the donor during the design of the M&E Plan.

Page 35: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 31

The design of the Program is logical. However as Program design documents note, reform through political means is not sufficient for equitable development. This is why these documents state that actions and advocacy by coalitions of women’s and gender-interested organisations on their own is equally important to achieve the Program’s Purpose and Goal. The private sector, however, is still not addressed here properly.

The MAMPU Theory of Change is also overviewed in Program design documents14. Figure 3 outlines the expected long-term, medium-term, and short-term outcomes of the Project, and the pre-conditions and interventions that are needed to be assessed regularly and managed where potentially controllable.

Figure 3

14 MAMPU Indonesia: Empowering Indonesian Women for Poverty Reduction, Program Design Document, Part A, September, 2012, AusAID, Section 1.1 and Figure 1.

Inputs Outputs –Activities & P ti i ti

Outcomes & Impact – ST, MT, LT

Multi-Stakeholder Participatory St t i

Priorities – 5 Thematic Areas

Staff, TAs, advisors, Funds for events, media, research etc.

What We Do - Needs Assessments, Capacity Development, Communications Support Network or coalition development & management

Who We Reach National, regional, district level women’s organisations, Women’s caucuses, Women Parliamentarians

Short, Medium & Long-term Results ST - Capacity and Readiness MT – Voice and Influence LT – Access to Services, Resources??? Improved services, livelihoods, and quality of life

EXTERNAL

ASSUMPTIONS

EVALUATION

Long-term Outcomes (5-8yrs+):

Access (to Improved Services) and Livelihoods

Medium-term Outcomes (3-5yrs): Voice and Influence

Short-term Outcomes (2-3 years): Capacity and Readiness for Collective Action – support to ‘change actors’, leaders and potential leaders

Pre-conditions –(i) strengthened organisations, (ii) networks with more participation and focus, (iii) working with the media, (v) evidence based decisions and actions; (vi) advocacy, engagement,

Pre-conditions–(i) influencing opinions of communities, decision-makers, and private sector practices, (ii) Changing communities, decision-makers and private sector minds and practices , (iv) Getting issues onto policy agenda, (v)…

Pre-conditions: (i) Policy implementation that protects women’s rights and promote gender equality, (ii) improved services delivery performance

Influencing Factors –

Positive and negative

Page 36: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 32

A key additional challenge related to the Theory of Change of the Program will be; (i) whether it can be shown that the influence exerted by strengthened partners on parliamentarians has contributed to changes in government policies, regulations, and services and (ii) whether this change has contributed also to access (to services??) and livelihoods for poor women in Indonesia. Addressing these issues will be critical for M&E of the Project.

The complexity of the environment of the Program is captured in Figure 2 in Program design documents15. Some 13 enabling factors and 21 inhibiting factors have been identified.

Managing relationships with partners whilst assisting to make them stronger; letting them take the lead and also supporting them to develop networks and coalitions which is something that is not part of their usual way of operating, and additionally working with parliamentary allies in a fickle political environment where alliances can be bought and sold, is obviously likely to be challenging. The Program design, however, allows for a longer than normal time period to achieve its medium to longer term aims. This is also why developing a Learning Strategy for partners, and learning for the MC will be an important aspect of MAMPU.

4.0 Working with Partners

4.1 Partner Working Relationships Program design documents and meetings with AusAID and STATT16, its MAMPU Technical Assistance contractor in the Interim Phase of the Program, all stressed the importance of developing a supportive and learning approach to the management of and capacity-building of local partners. They also noted that these relationships should be characterized by interventions that are:

• Participatory and build on partners’ strengths • Help partners recognize the importance of making decisions based on evidence, analysis,

followed by decisions made on the basis of their mandate, strengths and plans • Support short and long-term wins, and cycles of learning • Support interventions that bridge partners’ national to local divides • Aim for widespread change and sustainability

Program design documents are equally clear about the type of support that will be provided to partners17:

• Capacity development for groups within an organisation, and for groups within networks and coalitions rather than for individuals

• Provision of equipment that is necessary to make organisations more effective • Some core funding if organisations face particular constraints that affect their ability to

operate effectively • Leadership forums, national and international networking, and the mentoring of future leaders • Funding is not directly available to the GOI, although costs associated with their involvement

in the program may be covered. The expected working relationship between the MC and local partners is thus clear. 15 MAMPU Indonesia: Empowering Indonesian Women for Poverty Reduction, Program Design Document, Part A, September, 2012, AusAID, pp36-37. 16 STATT – Strategy, Analysis, Action, Transnational Trends, www.statt.net 17 MAMPU Indonesia: Empowering Indonesian Women for Poverty Reduction, Program Design Document, Part B, September, 2012, AusAID, p12, and 16-17.

Page 37: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 33

Program Design Document B also overviews the governance and management arrangements of other key stakeholders in the Program (Section 4). However some of these require clarification, in particular between the MC M&E Specialist and the Analytical and Research Unit (ARU).

4.2 Partner M&E Capacity Needs During the participatory design stage of the Program, AusAID identified and selected a number of women’s and gender-interested organisations to partner with and support18. These organisations are listed below and profiled in the Program design documents19.

1. PEKKA – Pemberdayaan Kepala Keluarga Perempuan (Women-Headed Household Empowerment)

2. KPI – Koalisi Perempuan Indonesia untuk Keadilan dan Demokrasi (Indonesian Women’s Coalition for Justice and Democracy)

3. Kapal Perempuan – Lingkaran Pendidikan Alternatif (Circle of Women’s Alternative Education)

4. Aisyiyah – women’s arm of Muhammadiyah

5. Migrant Care – Perhimpunan Indonesia untuk Buruh Migran Berdaulat ( Indonesian Association of Migrant Workers’ Sovereignty)

6. Komnas Perempuan – Komisi Nasional Anti Kekerasan Terhadap Perempuan (National Commission on Violence against Women)

7. ILO - International Labour Organisation

8. Yayasan BaKTI – Bursa Pengetahuan Kawasan Timur Indonesia (Eastern Indonesian Knowledge Exchange Foundation)

During the Interim Phase of the program a Technical Advisory Contractor, STATT, began providing support to and assessing the capacities of six of this group - Aisyiyah, BaKTI, Kapal Perempuan, KPI, Migrant Care, and PEKKA.

Key findings from assessments in these organisations, plus anecdotal reports from AusAID, identified that partners are at different stages of understanding and using M&E. They also observed all surveyed partners need help to:

• Understand the importance of evidenced-based actions

• Gather data and conduct effective analysis

• Make decisions and plans that build on analysis and their strengths

Further, Program design documents note that partner organisations will take a key role in the M&E of their initiatives. Since it is not yet known which partners have M&E systems integrated into their work plans or what type of evidence they are currently planning to gather in relation to their work plans assessing this situation in each partner organization will be a top priority of the MC M&E Advisor in the upcoming weeks20. From this understanding tailored M&E support can then be developed and provided.

18 MAMPU Indonesia: Empowering Indonesian Women for Poverty Reduction, Program Design Document, Part A, September, 2012, AusAID, Part B, Section 2.2, pp8-11. 19 MAMPU Indonesia: Empowering Indonesian Women for Poverty Reduction, Program Design Document, Part A, September, 2012, AusAID, Part B, pp 44-48. 20 Once partner work-plans are received and reviewed, and partners are met in their organisations by the Capacity Development and M&E Specialist the specifics of what needs to be done will become more apparent.

Page 38: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 34

5.0 Key Considerations for the M&E Plan

5.1 MAMPU M&E Responsibilities As noted above, assessing and developing basic planning and M&E capacity in partner organisations will be a priority for the MC M&E in the next 12 months. Therefore, it is anticipated that the responsibility for gathering and analyzing some of the data that is needed to report on the progress and achievements of MAMPU during this period may need to be shared with the ARU, the MC team of advisors, and possibly some short-term consultants. For example, in order to work effectively the MAMPU M&E Advisor, the MAMPU Capacity Development Coordinator and the Communications/Knowledge Management Specialist are planning to integrate their capacity development and support interventions as much as possible into partners’ work plans and planning needs. There still may be gaps, however, in partners’ data and knowledge relating to baseline data and realistic performance targets. It is envisaged that in this and other similar circumstances, the ARU or another group or individual may be contracted to capture this information. This issue and the contracting process will be discussed with AusAID and will be elaborated further in the M&E Plan.

5.2 Evaluation and CAPF Questions Table 1 overviews the critical indicators from CAPF21 and key evaluation questions that the MAMPU Program M&E Framework need to address. The former still require discussion and confirmation with AusAID, while the latter are confirmed.

Also included in Table 1 are key hypotheses related to the MAMPU Program which will be tested during the Program’s implementation. The ARU is responsible for the most significant of these (Table 1, Hypotheses 1 and 2). The third hypothesis relates to the effectiveness and efficiency of interventions to bring about the required changes and will be ascertained by the MC Team, in particular the Finance and Grants Manager in conjunction with the Team Leader and M&E Specialist.

Table 1: MAMPU CAPF Indicators, Key Evaluation Questions and Hypotheses

CAPF Headline Indicators Related to MAMPU Saving Lives 2 and 6 Number (x) of additional births attended by a skilled birth

attendant Percentage of water and sanitation committees in which women are equally represented

Promoting Opportunities for All 15

Number (x) of women survivors of violence receiving services such as counselling

Sustainable Economic Development 20, 21, 22,

Number (x) of poor women and men who increase their access to financial services* Number (x) of poor women and men with increased incomes* Number (x) of additional poor women and men able to access social transfers (such as cash or in kind transfers including food) *

Effective Governance 28 Number (x) of civil society organisations supported to track service provision

Key Evaluation Questions for MAMPU22

21 Headline Results and Indicators that ensure the Australian Government uses to ensure aid is delivered efficiently and effectively. 22 MAMPU Indonesia: Empowering Indonesian Women for Poverty Reduction, Program Design Document, Part A, September, 2012, AusAID, Part B, Section 5, pp28-32.

Page 39: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 35

Capacity and Readiness for Collective Action

Key Evaluation Question 1: How has the program affected the partners and networks' capacity to influence government reform?

Voice and Influence Key Evaluation Question 2: How have the partners and networks influenced government reform in relation to the needs and priorities of poor women?

Access and Livelihoods Key Evaluation Question 3: How has the program contributed to improved access and livelihood for poor women?

Context Key Evaluation Question 4: What is changing in the context, and how should the program respond?

MAMPU Hypotheses to Be Tested23 Responsibility ARU

That the multi-stakeholder approach (increased density and capacity of networks, and district-level participatory analysis to identify solutions, including women) is more effective than traditional approaches of expert or administratively identified solutions in achieving government reform that benefits poor women That use of an evidence base (including gender-disaggregated data) will increase the influence of CSOs on government reform

Annual assessment of costs (time, resources) & benefits (reduced duplication, ideas, improvements) of the approach.

That the increased communication between national partners through the program's design and management has positive benefits for the program’s outcomes.

5.3 Geographic Focus and Cross-cutting Issues The program does not have a specific geographic focus. As the Project Purpose states, those areas that MAMPU will operate in will be selected during Program implementation based on partner analysis on where they can make progress. The GOA also has clear over-arching policies that need to inform the design of the MAMPU M&E Framework and the frameworks of partners. These relate to gender, disabilities, environment and Disaster Risk Reduction, as well as child protection.

Gender equality and women's empowerment are already the foci of MAMPU and its partners. However partners may not have considered associated risks and opportunities related to the environment and climate change, child protection and disabilities.

As partner work-plans and strategies become evident, compliance issues related to these cross-cutting factors will be explained to partners and incorporated into their implementation plans and within the MAMPU M&E Framework and reporting.

Once this is completed, within twelve months of the start of the program, as required by AusAID, these compliance issues and associated proposed actions will be revised.

It is known that partners like Komnas Perempuan already have a mandate to address issues of violence against women with disabilities. BAPPENAS, the program’s host, plans to collaborate with this program on issues of identifying disability issues in the context of social protection. Other partner organisations have also expressed an interest in expanding this aspect of their work.

5.4 Capacity for Readiness, Voice and Influence

23 MAMPU Indonesia: Empowering Indonesian Women for Poverty Reduction, Program Design Document, Part A, September, 2012, AusAID, Part B, Section 5, p33.

Page 40: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 36

In order to identify changes in the way partners work during and after the Project – both as individual organisations, and as coalitions and networks of women’s and gender-interested organisations, it will be important to use a mixed multi-method approach to enable information to be gathered about their readiness and capacity for change, plus voice and influence.

Similar mixed multi-method data capture tools and techniques will also be needed to identify the outcomes and impacts of interventions of individual organizations, and coalition and networks of women’s and gender-interested organisations on policy, regulation, and service changes.

5.5 Reporting Requirements MAMPU M&E is responsible for managing and developing performance related reports against the Program’s M&E Framework. Those reports that will involve or be the responsibility of the MAMPU MC include Quality at Implementation Reporting, Annual Program Performance Reporting, Mid-term Reporting, and End of Project Evaluation.

Partners will be required to provide monthly grant disbursement reports to the MC, as well as six-monthly implementation reports. MAMPU MC specialists will ensure that partners understand and have support to complete these requirements.

5.6 M&E Budgeting As MAMPU is seeking to influence complex, complicated and largely unpredictable processes of change within a range of contexts, and with a diverse array of stakeholders, 10% of the total MAMPU Program budget has been allocated for M&E.

As program documents note this amount is significant, but it is not only to ensure accountability for donor funds, but also to support a central strategy to increase the effectiveness of the women's and gender-interested organisations and their networks.

This budget includes allocations for monitoring, evaluation, research, capacity development for M&E and learning activities.

Page 41: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 37

ANNEX 2: THREE POSSIBLE TOOLS FOR THE MAMPU PARTICIPATORY MONITORING TOOLBOX Ten Seed Technique

The Ten Seed Technique is a modified Participatory Learning and Action (PLA) tool. It was introduced by Dr Ravi Jayakaran (see Jayakaran, 2002) as a tool that can enable illiterate community members to participate in discussions about their community’s needs. It is useful in gathering qualitative information on various issues, especially related to the perceptions of the community and the way people see themselves in relation to others. Each group of participants is given ten seeds to allocate across categories that represent answers to a particular question. The group is asked to allocate the seeds: across several categories or to place all 10 on a single category. The discussion over the placement of the seeds often highlights important issues that reflect the views of the group participants. The Ten Seed Technique enables probing deeply into different dimensions of an issue.

Photovoice Photovoice or photo elicitation is a method that gives participants access to a camera to create their own images that represent their experience of the subject at hand. Participants are given a digital camera, and over a defined time period are encouraged to take photos of things that are important to them in relation to the sports program. These images are then used as a basis for conversation and discussion to develop a shared interpretation of the photos. This method also gives the participants autonomy in defining the aspects of change that are important to them, rather than responding to external ideas developed by participatory monitoring facilitators.

Most Significant Change

The Most Significant Change (MSC) technique is a form of participatory monitoring and evaluation. It is participatory because many project stakeholders are involved both in deciding the sorts of change to be recorded and in analysing the data. It is a form of monitoring because it can occur throughout the program cycle and provide information to help people manage the program. It contributes to evaluation because it provides data on impact and outcomes that can be used to help assess the performance of the program as a whole. The process involves the collection of Significant Change stories emanating from the field level, and the systematic selection of the most significant of these stories by panels of designated stakeholders or staff. When the technique is implemented successfully, whole teams of people begin to focus their attention on program impact, learning occurs through discussion and areas for improvement can be identified (see Davies and Dart, 2005).

Page 42: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 38

ANNEX 3: DRAFT METHODOLOGY FOR PARTICIPATORY ORGANISATIONAL CAPACITY ASSESSMENT

The concept of ‘capacity’ Evaluating changes in capacity has been the subject of considerable discussion, debate, and theorising in international development, resulting in a proliferation of approaches, methods, and tools. A lengthy discussion of this literature is outside the scope of this document. Nevertheless several distinct themes that have significance for the Program emerge from a brief review.

Firstly, there has been a steadily growing recognition that “the real impetus for systemic change comes from within organisations. It can be assisted, but not replaced by outside interventions. An effective M&E system must acknowledge and understand this” (Morgan, 1999). Consequently any process of evaluating capacity development must engage deeply with members of the organisation that is anticipated to develop. Put differently, evaluation of capacity development must be participatory.

Secondly, capacity develops because of change at several levels: the individual, the organisation, and the wider systems of organisations. An organisation may comprise highly skilled individuals who are nevertheless constrained by dysfunctional leadership or cumbersome procedures. This means that evaluation of capacity development should, as much as possible, seek to understand how capacity is changing in relation to each of these dimensions.

Thirdly, recent research by Engel et al (2008) and others points to the complex factors that interact to influence how the capacity and performance of organisations evolves. Capacity, according to this thinking, is an ‘emergent’ property of a system making it difficult and possibly counterproductive to apply overly predictive approaches to M&E. The European Centre for Development Policy Management (ECDPM) identifies five elements—or ‘core capabilities’—that interact to produce capacity and performance in different organisations:

1. the capability to survive and act 2. the capability to generate development results 3. the capability to relate 4. the capability to adapt and self-renew 5. the capability to achieve and maintain coherence.

These core capabilities can be simplified into four capabilities that are more intuitive and easier to use in a workshop setting:

• DOING – the organisation’s ability to carry out its main activities; • MANAGE – the organisation’s ability to administer and manage the resources

needed for these activities; • RELATE – the ability of the organisation to establish and maintain the connections

and relationships needed to carry out their work; and • CHANGE – the ability of the organisation to cope with and adapt to changes in the

context.

Page 43: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 39

The annual assessment at the national and district level could occur through a participatory organisational capacity assessment (POCA)24 that builds on this contemporary approach to capacity assessment. This process will be facilitated by a team comprising the CD Coordinator, the M&E Specialist and Partner Facilitator. The initial round of capacity assessments will serve as a qualitative baseline, with annual repeats.

A Participatory Approach to Assessing Capacity As discussed above, a participatory approach to capacity assessment is preferable for a range of reasons. A workshop-based approach provides a good opportunity to actively engage members of an organisation in assessing their own capacity. This section provides guidance on how to prepare for and facilitate a workshop for this purpose.

Before the workshop Required materials:

• Butcher’s Paper; • Marker pens; • Dot point stickers for ranking; • Laptop and projector for PowerPoint introduction.

Venue: The POCA process is intended to be flexible and deliberately moves away from the highly structured and hierarchical dynamic that exists in formal classroom or meeting to encourage open discussion. Keeping this in mind, it is important that the venue for the workshop:

• Is large enough for the number of participants; and • Has sufficient chairs and tables that can be easily moved around; and

Participants: The participants of the workshop should reflect the purpose. In this case, since the purpose of the POCA exercise is to assess capacity of partner organisations, participants should represent a cross section of roles in the organisation. Bear in mind that a maximum of approximately 20 participants is feasible with a facilitation team of 3.

Workshop Process This section describes the process for facilitating a participatory capacity assessment workshop. The initial assessment will provide a baseline of organisational capacity and will involve four main steps:

1. Set the scene; 2. Develop a capacity assessment framework tailored to the organisation; 3. Self-assessments by participants against each core capability; 4. Collate and analyse the results with the participants; and 5. Figuring out what action to take.

At each stage participatory techniques are important to encourage discussion and interaction between participants. This guidance proposes several such techniques, drawn from previous

24This methodology has been loosely adapted from widely used approaches to organisational assessment employed by organisations such as PACT’s Organisational Capacity Assessment Tool (OCAT—downloadable at http://www.pactworld.org/galleries/resource-center/Intro%20to%20OD%20First%20Edition.pdf), as well as the M&E approach suggested in ECDPM (2008), and the various approaches described by Simister and Smith (2010). This methodology has been specifically developed by Chris Chevalier and Stewart Norup of Sustineo Pty Ltd for use on the regional Australian Sports Outreach Program (funded by AusAID and delivered by the Austra

Page 44: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 40

applications of this process in the Pacific. Nevertheless, these techniques are suggestions only and experimentation with different methods is definitely encouraged. STEP 1: Set the scene At the risk of stating the obvious, begin the workshop by explaining the purpose of the exercise, introducing key terms and concepts.25 A brief PowerPoint presentation can be useful aid to this process. Core issues to present in this introductory session are:

• Purpose of the session and the agenda; • ‘Capacity’ and ‘capacity assessment’ – some basic

points to explain these concepts. Particularly important points include the idea that has multiple levels (i.e. individuals, organisations, systems of organisations), that changes in organisational capacity need to be driven from within not by outsiders, and that evaluating or assessing capacity should be participatory;

• Some basic ‘ground rules’, for example, respect for all views, and keeping discussion on track.

STEP 2: Develop the capacity assessment framework The main idea behind a participatory approach to capacity assessment is that members of an organisation develop their own assessment framework.

The starting point for this will be the collective identification of ‘pointers’ – akin to indicators. This process will start by introducing four core capabilities: DOING; MANAGING; RELATING; and coping with CHANGE. These four capabilities, it should be explained, underpin capacity in any organisation.

Using a participatory process (see box 2), participants in each workshop will be asked to identify specific practices – these are listed under ‘DOING’. Participants then identify specific elements under MANAGING that relate to community policing – these include administration, finance, and logistics. Similarly, they identify the groups and organisations that they RELATE to in undertaking this work, and the major changes that they face in the coming 12 months (CHANGE).

Following this step, participants sort all the comments noted under each capability into 3 or 4 score-able areas – ‘pointers’. Once this step has been completed, participants have created an assessment framework for their organization.

Once the capacity assessment framework has been developed it is important that all participants are given the opportunity to review it to ensure that all understand what each

STEP 3: Refining the ranking scale

25 Again, at the risk of stating the obvious, any participatory workshop should also start with a thorough introduction, not only of the facilitators but also of the participants to each other.

Suggested Technique: ‘Table Talk’/‘World Café’. This highly portable method of consultation involves the use of tables, each with a question relating to particular area of interest (on Butcher’s Paper). In this instance, it is suggested that each table be devoted to one of the four core capabilities: Doing; Managing; Relating; and Coping with Change. Participants are invited to circulate among the tables noting down their responses to each question on sticky notes. This makes the notes easy to sort into themes.

Page 45: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 41

The second step involves the discussion and refinement of the scale that will be applied for ranking the organisation against each pointer that links to the core capabilities identified in Step 1. The recommended approach will be a ten-point scale along the lines of the following:26

Figure 5: Scale for rating 'pointers'

Rating Interpretation

10 This element of capacity is the best that it could be 5 Existing capacity has some strengths and some elements that could be

further strengthened 1 There is no capacity in this area

STEP 4: Assessing the partner against each core capability The facilitator encourages discussion around each core capability and pointer to explore the current capacity of the partner organisation at that particular point. Discussion will be followed by rating the organisation using a voting process to facilitate assessments against each ‘pointer’. One possible approach is dot point voting, where each workshop participant is allocated one ‘vote’ (in the form of an adhesive sticker in the shape of a coloured dot) to distribute along the scale (for each capability), which could be presented on paper or on a white board. This process will assist in providing a clear indication of the balance of views and will stimulate discussion on any emerging differences. During the endline assessment, the scores from the baseline will be revealed to stimulate discussion on what may have changed, why and how this has occurred, and the extent to which it may have changed. This approach does not presume to attribute the changes to the specific support provided by the organisation, but instead seeks to explore any potential ‘plausible associations’ through a discussion of the timing of various key events, be they support from MAMPU or other contextual factors. Similarly, factors that hinder the process of change in the core capabilities will be discussed as part of this session.

An alternative process where discussion may be difficult is to ask participants to allocate a score between 1 and 10 to each ‘pointer’ that has been developed. This is done silently using a score sheet handed to participants. The results are placed on a chart on a wall with each participant’s name and corresponding scores (participants are told that their score will be public). The differences and similarities by individual scores are then used by the facilitator to generate discussion around the core capabilities.

STEP 5: Collating and analysing the results and considering the implications for action Much of the analysis inherent in this approach is carried out by participants as part of the facilitated workshop process. The final session of the POCA will seek to move participants from reflecting on organisational capacity to the implications for action. For example, the POCA ranking may highlight that there is scope to improve. Specific actions will be identified that can support this process and any specific assistance that can be provided by the MC. The results will be summarised in written form by the M&E Specialist.

26 Previous experience with organisational capacity assessments suggests that scores could decrease as an organisation gains more knowledge about a capability and makes a more accurate assessment of their capacity. Nevertheless, the discussion surrounding this eventuality will be of value in generating insight into the changes experienced by an organisation.

Page 46: MAMPU: Empowering Indonesian Women for … Empowering Indonesian Women for Poverty Reduction ... 4.6 Special Research Studies ... (Empowering Indonesian Women for Poverty Reduction)

MAMPU Monitoring and Evaluation Plan 42

The POCA is expected to require up to one day for each unit of the PNTL. The process will be repeated each year to engage members of each organisation in a process of reflecting on changes in capacities perceived from year to year. This exercise will involve an annual repetition of Steps 2 to 4.

While the methodology will generate a set of numerical scores each year, the real value of the process will be the collective deliberation, reflection and discussion that the process enables. The scores themselves should not be regarded as robust or comparable across districts (although broad patterns should be discernible from a comparison of different districts). Rather, they provide a rough indication of perceptions of capacity along with collective insights how and why these may have evolved over time.