Embed Size (px)
Citation preview
1
The Independent Monitoring and
Evaluation Project for the State Level
Programmes (IMEP)
GEMS Lesson Learning Review
Final Report
Author(s)
George Abalu
Emmanuel Adegbe
Gulden Bayaz
Stuart Pettigrew
Melanie Newman Wilkinson
Abuja/Adelaide, 30th November 2015
2
Funded by UKaid from the Government of the United Kingdom.
This material has been funded by UKaid from the UK Government; however the views
expressed do not necessarily reflect the UK government’s official policies.
ECORYS
Ecorys UK Ltd Queen Elizabeth House 5th Floor 4, St Dunstan’s Hill London EC3R 8AD
Email: [email protected]
Web: www.ecorys.com
3
Table of Contents
TABLE OF ACRONYMS………………………………………………………………………………… 4
1 EXECUTIVE SUMMARY .......................................................................................................... 6
1.1 Strategic Frameworks ............................................................................................................................. 7
1.2 Tools and Approaches ............................................................................................................................ 7
1.3 Management and Oversight ................................................................................................................... 8
1.4 Value for Money ..................................................................................................................................... 9
1.5 Impact .................................................................................................................................................... 9
1.6 Sustainability .......................................................................................................................................... 9
1.7 Summary of Recommendations to DFID ............................................................................................... 10
2 INTRODUCTION ..................................................................................................................... 12
2.2 Context of DFID Support in Nigeria ....................................................................................................... 13
2.3 Purpose and Scope ............................................................................................................................... 14
3 METHODOLOGY AND APPROACH ................................................................................... 15
3.1 Mid Term Review and Lesson Learning Review .................................................................................... 15
3.2 Data & Information Sources ................................................................................................................. 17
3.3 Review Team’s Approach ..................................................................................................................... 20
4 FINDINGS AND LESSONS LEARNED ................................................................................. 21
4.1 Formulation of an Appropriate Strategic Framework ........................................................................... 21
4.2 Tools and Approaches .......................................................................................................................... 25
4.3 Management and Oversight ................................................................................................................. 36
4.4 Value for Money ................................................................................................................................... 44
4.5 Impact .................................................................................................................................................. 50
4.6 Sustainability ........................................................................................................................................ 55
5 CONCLUSIONS AND RECOMMENDATIONS .................................................................... 59
ANNEX 1: SUMMARY PROFILES OF THE GEMS PROJECTS ............................................... 63
ANNEX 3: STAKEHOLDERS, GOVERNANCE AND PROCESS ISSUES ................................ 67
ANNEX 4: MTR AND LESSONS LEARNING EVALUATION FRAMEWORK....................... 70
ANNEX 5: PROJECT DOCUMENTATION USED FOR REVIEW AND ANALYSIS .............. 71
ANNEX 6: LESSON LEARNING WORKSHOPS – OUTLINE ................................................... 77
4
Table of Acronyms
A2F Access to Finance
ATA Agricultural Transformation Agenda
BDS Business Development Service
BIR Bureau of Internal Revenue
BMO Business Membership Organisation
CAC Corporate Affairs Commission
CBN Central Bank of Nigeria
CF Challenge Fund
CGAP Consultative Group to Assist the Poor
COO Certificate Of Occupancy
DCED Donor Committee for Enterprise Development
DFID Department for International Development (UK)
EEG Export Expansion Grant
ENABLE Enhancing Nigerian Advocacy for a Better Business Environment
FCT Federal Capital Territory
FF Feed Finishing
FFVDAN Fresh Fruit and Vegetable Dealers Association of Nigeria
FGN Federal Government of Nigeria
FLG Finished Leather Goods
FMARD Federal Ministry of Agriculture and Rural Development
FMITI Federal Ministry of Industry, Trade and Investment
FTE Full Time Equivalent (job)
GBP British Pound
GEMS Growth and Employment in States
IGR Internally Generated Revenue
IMEP Independent Monitoring and Evaluation Project
IPB Investment Promotion Bureau
KASTU Kano State Traders’ Union
KPI Key Performance Indicator
LGA Local Government Authority
LAPAN Leather and Allied Products Association of Nigeria
MAN Manufacturers’ Association of Nigeria
MDA Ministry, Department, Agency
MEDA Micro Enterprise Development Agency
MOU Memorandum Of Understanding
5
MPB Ministry of Planning and Budget
MSME Micro, Small and Medium Enterprise
M&E Monitoring and Evaluation
M4P Making Markets Work for the Poor
NGN Nigerian Naira
NGO Non-Governmental Organisation
NIRSAL Nigerian Incentive based Risk Sharing Agricultural Lending
OECD Organisation for Economic Co-operation and Development
PFI Policy Framework for Investment
PIP Performance Improvement Plan
PMQA Planning, Monitoring and Quality Assurance
P/NP Poor / Non Poor
PPEM Public-Private Engagement Mechanism
PPD Public-Private Dialogue
PPP Public-Private Partnership
PSD Private Sector Development
RMS Results Measurement System
RM Results Measurement
SLP State Level Programme
SME Small and Medium Enterprise
SWG Sector Working Group
TOC Theory of Change
TOR Terms of Reference
UK United Kingdom
VFM Value for Money
WBG Wholesale Buying Group
WEE Women’s Economic Empowerment
WRMS Wholesale and Retail Market Systems
6
1 Executive Summary
In May-June 2014 the Independent Monitoring and Evaluation Project (IMEP) was
commissioned to undertake an interlinked Mid-Term Review (MTR) and Lesson
Learning Review of the DFID-funded GEMS components in Nigeria.
A Framework was developed to provide the analytical structure for the MTR and
Lesson Learning Review (Annex 4). For the Lesson Learning Review, the reviewers
loosely based their methodology and approach around a cluster evaluation approach.
The cluster evaluation approach helps identify commonalities across the GEMS
components, but it also permits the necessary flexibility in areas for analysis given that
the projects are at different stages.
This report is a compilation of the lessons learned from GEMS 1, 3 and 4, and those
gathered from the GEMS 2 lesson learning review conducted in December 2013.
Learning themes span project relevance, effectiveness, efficiency, impact and
sustainability and the appropriateness of the Making Markets Work for the Poor (M4P)1
approach in the different GEMS contexts. The practicalities of applying M4P principles
and methodology in complex and dynamic markets and how particular GEMS
components are affecting systemic change in markets are also guiding themes to this
Review.
Certain important factors have influenced the approach for both the MTR and the
Lesson Learning Review. Firstly, the review team took into account the different stages
of implementation of GEMS and what could realistically be expected in terms of data
collection, analysis and learning across each of the respective projects. A second
important consideration was the fact that GEMS 3 is not technically an M4P project.
GEMS 3 has developed and applied its Business Environment Improvement
Framework (BEIF) as its overarching approach across the project. BEIF is described
by GEMS 3 as a system which is used to catalyse changes in states’ business
environments. The design of the framework ensures that reforms are demand driven,
cost effective and deliver sustainable change.2
In the case of GEMS 3, therefore, the focus for lesson learning was not M4P in
practice, but how the project is able to contribute to M4P learning and broader market
system development programming.
The Lesson Learning Review was based around 6 Thematic Areas, and these are
summarised below:
1 Detailed information and links to a wide range of resources detailing the approach and tools available to M4P projects can be
found at http://www.enterprise-development.org/page/m4p 2 More information on this approach and its development can be found at the GEMS 3 website: http://gemsnigeria.com/gems-3/
7
1.1 Strategic Frameworks
A sound Strategic Framework is the basis for successful implementation of any
programme. For this to be achieved, both DFID and the service provider need to be
clear at the outset on the agreed vision for a target sector, and consequently the
purpose of the project and/or interventions. As a follow on, the developmental
approach the project will adopt needs to be clearly communicated.
The biggest risk to successful implementation is a lack of clarity or agreement on
targets or approach, and this will inevitably lead to delays in implementation and
difficulty in achieving targets.
The process of applying the M4P approach from the outset will help projects with
respect to putting core programming elements in place. This process begins with
having a clear vision for the sector and for target beneficiaries in that sector; this in
turn requires clear understanding between DFID and the service provider on the
overarching rationale for the project (including its positioning in regard to other projects
and programmes) and the direction it should take.
What was in evidence from all 4 GEMS projects was the importance of a strong
knowledge of the target sector. Credibility is vital to position the project correctly, and
to attract the right partners. Credibility starts from using experienced staff, and requires
conscientious efforts and expertise in the early stages of implementation.
1.2 Tools and Approaches
The success of the GEMS projects shows that good intervention design is achieved
through strong stakeholder analysis and deep understanding of all the market players.
Market analysis needs to be front-loaded to ensure project strategy and early stage
interventions are well grounded.
In developing its intervention portfolio, a project needs to have a mix of intervention
strategies and partners to drive ‘quick wins’ for direct impact on the poor, balanced
with a selection of what are likely to be longer, slow-burn initiatives (e.g. that involve
policy or institutional change).
Projects should recognise the risks related to relying on a single partner across
interventions or at the core of an intervention area. Whilst initially a project may have
no choice but to work with one partner, but the project’s intervention strategies should
include opening up opportunities for more players in due course.
A market system development programme will invariably use a mix of tools and
approaches when applying an M4P methodology. Action research is a key tool for
attempting to bring new partners together, prove a business case, and for deepening
understanding and testing the market as to what can work. It should also contribute to
projects establishing credibility in their target sectors.
In analysing the GEMS programme, it is clear that using a mix of tools and approaches
to research, analyse and design intervention plans is not at odds with an M4P
8
approach, but in fact one of its strengths. Adjusting interventions to account for market
distortions, rather than trying to ‘fix’ the distortion is the preferred approach.
A clearly defined intervention logic (or business case) helps keep activities focused on
the target beneficiaries, and should be robust enough to guide intervention decision-
making - including timely decision-taking for dropping an intervention or partnership.
It also provides a path to an exit strategy. The counter to this is that the continuous
management cycle of designing, re-examining and revising intervention logics based
on unfolding results draws significantly on project resources.
What is important is the capacity for project management and field staff to continually
apply the M4P lens to shape market analyses, action research initiatives, partnership
choices and agreements, and what needs to be monitored and measured over time.
Applying an M4P lens should also sharpen a project’s ability to continually ‘read’ the
market (what is emerging in terms of changes in practices, behaviour, roles, attitudes,
etc.) and support accordingly.
Above all, it needs to be recognised that systemic change takes time and partnerships
and interventions will inevitably evolve throughout the project’s timeline.
1.3 Management and Oversight
The significant lesson is that a combination of qualitative and quantitative indicators is
required to ensure projects effectively capture systemic change. Hindering this though
is the focus on meeting (contractually required) numerical targets. The use of ‘stories’
as part of the reporting is an effective way to communicate results in combination with
quantitative indicators, adding depth to the numerical results.
All GEMS still utilise a log frame, but to be a useful tool in managing and delivering
M4P programmes, this has to evolve with the project. Revisions have to make sense
in the context of M4P. Furthermore, the log frame, Theory of Change and intervention
logics also need to be kept aligned throughout the project lifecycle in order to serve as
meaningful project management tools. It would also be useful to have a documented
process to support log frame revisions.
The complexities of measuring and managing M4P programmes justifies the use of
more innovative tools and approaches such as economic modelling and commercial
benchmarking to deepen understanding of business practices, business/sector
performance, and opportunities for scaled impact.
Development of frameworks for cross programme collaborations should not be overly
prescriptive, in particular given that M4P projects are likely to evolve at very different
paces, and with varied degrees of success across interventions. Such frameworks or
agreements require clear input from funding bodies (or neutral partners) in framing
and managing deeper cross project collaboration or integration. Essential to this
collaboration though is an appropriate funding mechanism that provides timely
financial and technical support.
9
1.4 Value for Money
A strong focus was given to Value for Money analysis in the Mid-term Review, and a
number of lessons could be drawn from this experience.
VFM measures ideally need to be framed from the outset as a management and
measurement tool. The ability to interpret the value of a project originates from the
project’s Theory of Change, which should include robust and well-evidenced
intervention logics.
Assessing VFM is a continuous, evolving process of comparison with cost benchmarks
and current practice and seeking/achieving improvements to these. Even though
comparison across a broad range of projects may sometimes diminish its utility,
external benchmarking research and analysis can still be useful.
The GEMS projects also showed that achieving equity goals has cost implications, as
demonstrated by their pro-poor and women reach results and associated costs.
Two major challenges for VFM in M4P programmes come from the often extended
timescale to achieve impact at scale, and attribution to show causality in a complex
operating environment.
A variety of methods need to be used together to address attribution with regard to
outcome and impact: some examples include using proxy indicators to capture
systemic change, triangulation of quantitative findings and estimates through
qualitative studies, and attribution through key stakeholder feedback.
1.5 Impact
With any systemic change, there is the potential for unintended positive or negative
impacts to occur. It is the role of projects to understand and respond to these, and
ultimately successful M4P intervention strategies are those that are driven by the
needs of market place actors. Projects need to know when to get out of the way!
As interventions unfold, projects need to be able to sharpen their lens with respect to
how and what they monitor so unintended outcomes are identified and appropriate
responses made early on. Project teams have to have the ability to remain flexible and
responsive to developments in the market place. This starts with project management
style, but even more critical is the right skills sets and expertise at field level.
To maximize impact, scaling up needs to be based on ‘new’ intervention plans that
adequately reflect the context (recognising new needs and circumstances) - not just
repeating a successful model. Therefore rigid models or over standardization of a
model or toolkit will lead to replication of weaknesses or an inability to respond to
changes in the market system.
1.6 Sustainability
Stimulating and enabling actors to improve the delivery of goods and services to the
poor is at the heart of the M4P approach. Free market forces do not always benefit the
10
poor, so it is important for projects to test that changes in the market system do not
exclude the poor.
Whilst adjusting strategies as project develop is essential, too much adjustment in
response to market constraints may be a weakness in achieving sustainable impact.
Asking why strategies are being adjusted and what is triggering these adjustments is
what matters.
Sustainable, systemic change requires projects to select and build partnerships in
which there are senior individuals with the right orientation and who appreciate the
opportunities for tangible gains on offer from the projects.
An emerging learning theme is the need for programmes like GEMS to have much
longer term visions for not only the sectors they are working in but for the communities
they strive to transform. Such an approach involves looking at the ‘bigger picture’ of
what is envisaged across sectors as a result of an M4P intervention.
1.7 Summary of Recommendations to DFID
1. Agreed Vision (or Theory of Change) is Essential: Donors and service providers
need to agree on a clearly articulated vision for the target sector(s), or Theory of
Change, at the outset of the programme.
2. Front Loading of Market Analysis: Programme design must be informed (and
developed) based on sound market analysis. Donors must allow projects to
undertake adequate market analysis early, so that interventions can be properly
designed.
3. Use of Qualitative Indicators in log frames/reporting: Project management log
frames should not only include quantitative indicators, but also qualitative
indicators that reflect systemic changes and impacts.
4. Annual Reviews: DFID should also consider how the Annual Review process
(including content of the ARIES template) could be more useful to projects.
5. Cross Programme Collaboration: The GEMS programme was conceived as a suite
of interacting projects. However, a lack of coordination in the log frames meant
collaboration was less effective.
6. Use of DCED Standards: Irrespective of the approach being taken, the use of the
DCED standards for results measurement is a sound basis for assessing impact.
7. Results Measurement (the Handbook): The GEMS projects were intended to use
a standardized approach to results measurement, ‘the Handbook’. If this approach
across a suite of programmes is to be used, more attention should be given to
ways to reflect on the usefulness of the handbook and to review and upgrade the
approaches.
8. Responding to Unintended Outcomes: Whether positive or negative, unintended
outcomes or results are likely to occur over the course of a programme. Allowing
the flexibility in the initial project design to respond to these is important.
11
9. Use of PPEM & PPD: The public private engagement mechanism (PPEM) and
public private dialogue (PPD) approaches applied by GEMS 3 are both successful
components of the project’s strategic framework and would be useful in the context
of other market system development programmes.
10. Embedding technical expertise with partners: Both GEMS 1 and 3 effectively used
the model/approach of embedding technical expertise from the project with
partners. There are aspects of this approach that has helped the projects succeed,
and which would be useful learning for a broader market system development
programme audience.
11. Innovative Tools and Approaches: Donors, including DFID, should support
research and development for introducing more innovative tools and approaches.
Successful projects will use a range of delivery mechanisms, and not be restricted
to working within one approach only.
12. Value for Money: With respect to VFM and impact, the use of ratings and
weightings could be further examined and considered by DFID Nigeria, and other
donors, as a means to address the challenge of demonstrating results and VFM
while value takes time to generate.
12
2 Introduction
The Growth and Employment in States
(GEMS) programme is £195 million
programme, jointly funded by a World Bank
loan of £105 million and DFID Nigeria grant of
£90 million. The programme aims to increase
growth and employment across six sectors of
the Nigerian economy, as well as through
business environment reform.
The DFID-funded sector portfolio to-date
comprises:
GEMS 1 - Meat and Leather, which began in
2010 and is due to close in 2015, hence
effectively entering its final implementation
phase;
GEMS 2 - Construction and Real Estate, which
began its activities in April 2010 and ceased
operations at the end of December 2013,
following DFID Nigeria’s assessment that the
project was unlikely to achieve its objectives
within the project lifetime;
GEMS 3 - the Business Environment, which
began in August 2010 and is due to close in
2015, so is effectively entering its final
implementation phase; and
GEMS 4 - Wholesale and Retail Commerce,
which began in August 2012, is due to close in
2017, and is in its initial implementation phase.
Annex 1 provides summary profiles of the four GEMS projects.
2.1 Applying the M4P Approach
Each of the GEMS projects applies market system strengthening approaches to
achieve market-led and pro-poor impact. Though working in very different sectors and
contexts, programme partnerships and interventions reflect the ethos of the ‘Making
Markets Work for the Poor’ (M4P) approach. M4P aims to tackle fundamental
weaknesses in market systems to effectively contribute to systemic, large-scale and
sustainable changes that positively impact the poor.
What is M4P?
The ‘Making Markets Work for the Poor’ (‘M4P’) approach aims to tackle fundamental weaknesses in market systems (which are the backbone of the Nigerian economy) to effectively contribute to systemic, large-scale and sustainable changes that positively impact the poor by improving their livelihoods, and consequently reducing poverty.
At its core, M4P is a Private Sector Development approach, working with the market system and identifying actors who are able to affect change at the market system level, rather than targeting interventions at specific individuals.
A range of tools are used within M4P, with the target being market driven change. This requires projects to identify weaknesses, and then facilitate market actors to develop a ‘business case’ around the changes needed to alter the markets behaviour. A measure of success in this approach is to see other actors ‘crowding in’ to the market – namely copying and adapting approaches remotely from project activities. M4P is a broad approach, and may include other approaches such as value chains and business development service.
Detailed information and resources related to M4P can be found on the DCED website: http://www.enterprise-development.org/page/m4p
13
The Department for International Development (DFID) is making significant
investment into M4P as a programmatic approach centred on pro-poor growth and
inclusiveness. It has a wide spectrum of programmes globally with varied budgets,
project life times, and across different country contexts and sectors. These
programmes, including the Nigeria GEMS components, have a wealth of knowledge
and experience, which DFID is able to tap into and share with the broader community
of funders and practitioners involved in market system development programming.
The core elements of an M4P approach are set out in the box above. As experience
with M4P develops further, there are also debates on strengths and weaknesses of
the approach, including the extent to which the poor may benefit from market change.
A helpful discussion of some cautions and caveats can be found in “A Synthesis of the
Making Markets Work for the Poor (M4P) Approach”3.
As noted later in this report, GEMS 3 moved away from using a M4P approach, as it
was seen to be less well aligned with the business environment outcomes being used
by GEMS 3. In this context, GEMS 3 developed the Business Environment
Improvement Framework (BEIF) as a tool to interact with partners. BEIF still includes
one of the fundamental approaches of M4P which is to work with the actors to identify
systemic change that can be achieved through them, not on their behalf.
2.2 Context of DFID Support in Nigeria
Although the Nigerian economy is growing rapidly (it was recently classified as the
largest in Africa), this growth is not trickling down to the poor. Nigeria continues to see
deep levels of poverty right across the country, with rural areas also heavily affected
by this, and a lack of economic opportunities. Nigeria is classified by the World Bank
as a lower middle-income country, but also reported as having 46% of the country
living in poverty, indicating the issue of uneven distribution of wealth.
Whilst GDP growth has been strong over recent years (and a recent re-balancing of
the GDP calculation raised Nigeria to Africa’s largest economy) the growth rates are
reducing quickly.
Despite bright spots in industries such ICT, entertainment, and construction, the oil
sector continues to dominate the economy, accounting for over 90% of exports and
75% of consolidated budgetary revenues. Nigeria ranks poorly in areas such as
literacy in rural areas, especially among women and adolescent girls, and according
to United Nations Development Programme (UNDP) international human
development indicator (HDI) ranking. Unemployment is high, estimated at 60%
amongst the youth, and equally high in rural areas. There is an estimated 50 million
under-employed youth. As much as 92% of the population is in informal employment.
3 A Synthesis of the Making Markets Work for the Poor (M4P) Approach, DFID and SDC, 2008, p.23 http://www.value-chains.org/dyn/bds/docs/681/Synthesis_2008.pdf
14
The ability to engage in business activities in Nigeria remains difficult with respect to
the insecurity in the country, in particular across the north, as well as across a range
of doing business indicators. According to the Economist Intelligence Unit (EIU),
Nigeria ranks 76th out of 82 countries. Regionally the situation is no better, with Nigeria
ranking 13th out of 17 neighbouring / regional countries. According to the WB ‘Doing
Business’ ranking for 2014, Nigeria is 147th out of 187 countries, and dropped nine
places from 2013 to 2014. All of this shows the difficulty enterprises have in creating
the employment and income opportunities needed to eradicate poverty.
The country’s wholesale and retail sector, which is essentially traditional in operation,
comprises one fifth of the Nigerian economy and directly or indirectly impacts the
livelihood of almost 170 million Nigerians. Consequently, interventions that can open
up more opportunities for low income groups and women that are key players in the
country’s wholesale and retail markets, are likely to contribute significantly towards
improved incomes and ultimately poverty reduction.
Land, tax and investment systems are permeated with lengthy bureaucracy, lack of
transparency, and corruption. There is unwieldy overlap and duplication of taxes
across Federal, State and Local Governments. Unorthodox means of tax collection
are commonplace and women in particular have reported feeling harassed and
threatened in their workplaces. Business owners face complex and irregular policies
and regulations when trying to register their business or gain title to land or other forms
of business certification. Historically there has been very little dialogue between
government agencies and the private sector. As a result policies, relationships and
even reform strategies frequently stifle rather than incentivize investment and growth.
Gaps in governance and leadership capabilities particularly at local government level
also slow down reform in the business enabling environment.
The government of Nigeria has recognised the potential for agriculture to create
income and employment, and has embarked on a wide-ranging Agricultural
Transformation Agenda (ATA). Meat and leather is identified as one of the industries
that can create opportunities, in particular in increasing incomes in rural Nigeria.
Apart from potential insecurity surrounding the elections, major changes in
governments will also lead to changes in the priorities and current programmes. The
lead up to elections also tends to be accompanied by a reduced willingness for elected
officials to make decisions on new programmes, or implement activities that may
marginalise any part of the electorate. As the GEMS suite of projects focus activities
through state based agencies, this may have a significant impact.
2.3 Purpose and Scope
In May-June 2014 the Independent Monitoring and Evaluation Project (IMEP) was
commissioned to undertake an interlinked Mid-Term Review (MTR) and Lesson
Learning Review of the DFID-funded GEMS components in Nigeria, both individually
and as a collective. Annex 2 contains the Terms of Reference for this.
15
The purpose of the MTR was to provide a structured assessment of GEMS 1, 3 and 4
effectiveness by examining relevance, efficiency and effectiveness across core
programmatic elements. This included efficacy of operational interventions and
progress to date, as well as the likely future trajectory for each project in their
remaining lifetimes and their potential to contribute to future sustainability.
The primary rationale for the second component of the process, the Lesson Learning
Review, is to contribute to learning on market systems approaches, and the ‘Making
Markets Work for the Poor’ (M4P) approach in particular, in order to improve future
application of M4P either in Nigeria or elsewhere, and to share learning from GEMS
project experiences in Nigeria with the broader community of practitioners working on
M4P projects and approaches.
This report is a compilation of the lessons learned from each of GEMS 1, 3 and 4, and
those gathered from the GEMS 2 lesson learning review conducted in December
2013. All the lessons presented have particular relevance for contributing to know-how
and experience of implementing the M4P methodology that is core to the GEMS theory
of change and intervention approach.
Details on the key stakeholders, governance and process issues, including
dissemination plans following the Review are described in Annex 3.
3 Methodology and Approach
3.1 Mid Term Review and Lesson Learning Review
The Mid Term Review and the Lesson Learning Review were conceived as two parts
of an overall, interlinked review process. They combine to offer scope for in-depth
review, analysis, and cross comparison of the GEMS projects.
The technical approach and methodology were developed to support robust and
credible data collection, analysis and reporting. Whilst the MTR can be considered as
the ‘confirmatory’ part of the Review – applying a deductive approach to test the rigour
of the theory of change and verify the evidence of performance and results, the Lesson
Learning component can be considered as more exploratory - applying an open-ended
‘inductive’ approach to compare, contrast and draw out lessons and principles on what
works in market system development.
It is important to note that the methodology employed and information generated from
the MTR is also used to inform this Lesson Learning Review.
As each of the four GEMS components has a different focus and offers different
experiences with respect to their project lifetimes, the aim of this Review is not to make
direct comparison of the projects or delivery models. Rather, this Review is a synthesis
of the lessons learned across the suite of projects.
16
The Reviewers loosely based their methodology and approach for gathering and
analysing learning around a cluster evaluation approach. Cluster evaluation is a
means of identifying learning across a portfolio of projects that share some similarities,
but also a number of differences. The approach allows us to look across a group of
projects to identify commonalities for success (or failure) that under closer scrutiny
take on greater significance4.
Similar to the cluster evaluation approach, the key characteristics of the Lesson
Learning Review are that:
It is outcome oriented, seeking to draw out key factors contributing to performance
and outcomes across programme design, management and delivery and where
possible identify common themes and learning that arise;
It builds on the MTR process (in other words the findings from the MTR provide the
basis for learning) and has the intention to provide learning that furthers our existing
knowledge basis on M4P programming;
It has been highly interactive and participative in nature, requiring cooperation,
communication and collaboration of the GEMS projects, partners and
stakeholders;
It provides scope for strong formative as well as summative analysis, and can
provide guidance for the projects while their initiatives are still under way, as well
as drawing conclusions about overall delivery;
It has been open to emerging findings (including unexpected findings), which
is well suited to the nature of the M4P approach.
Whilst the MTR analysis process confirms to what extent change has occurred, the
Lesson Learning Review provides collective analysis of the circumstances that have
led to that change, and what contextual and programmatic factors have contributed to
varying degrees of success. The intention of this process being to gather insights and
learning that can be applied in the GEMS projects as they continue to implement, and
in the broader context of M4P and market system development programming.
The cluster evaluation approach helps identify commonalities across the GEMS
components, but it also permits the necessary flexibility in areas for analysis given that
the projects are at different stages. In the case of GEMS 1 and 3, for example, there
is scope for learning about impact and sustainability. GEMS 2 analysis focuses on
learning from the review conducted in December 2013, which examined various
aspects of GEMS 2 design, management and delivery. In the case of GEMS 4, it is
understood that it is too early to explore learning on impact and sustainability
4 ‘Overall evaluation of the M4P delivery model: proposed approach’ - Specialist Evaluation and Quality Assurance Service,
Service Request Report; Burt Perrin, IOD PARC, November 2013.
17
specifically. However, we are able to learn from GEMS 4 tools and approaches for
market analysis, intervention design and partner selection, stakeholder engagement,
M&E Framework design and application, and how these are contributing to shaping
project relevance and effectiveness.
The planned approaches for both the MTR and the Lesson Learning Review lend
themselves well enough to cross comparison and case studying of GEMS 1, 3 and 4.
At the same time, the MTR and Lesson Learning Review will offer the opportunity to
add to the existing body of knowledge on what works and what does not work in M4P
programming, whilst drawing conclusions on what typifies a good market system
development approach based on the evidence provided so far by the GEMS projects.
An overall Review framework was developed to provide the analytical structure for the
MTR and Lesson Learning Review with respect to the core thematic areas of study,
including pre-formulated sets of questions, which were used to guide data collection
and analysis for both the MTR and the lesson learning. The Evaluation framework is
contained as a separate document (Annex 4).
3.2 Data & Information Sources
The following sources of information and data were used to inform this Lesson Learning Review.
3.2.1 Programme documentation and data
The Reviewers drew heavily on programme documentation and data that has been
collected / produced internally by the GEMS projects over their respective project
lifetimes. Annex 5 contains the detailed list of documents that were submitted by the
GEMS project teams. Documentation was requested well in advance of the review
process and included: design documents - such as original terms of reference,
inception reports and formal inception reviews, market analyses and studies used to
underpin the strategic framework; project management documents – such as M&E
plans, latest iteration of the log frame, selected results chains / intervention logics; and
research, surveys or studies that have informed implementation – such as baseline or
end line surveys, gender research, value chain analyses5, and market monitoring and
impact assessment exercises, and project quarterly and annual reports.
3.2.2 GEMS 1, 3 and 4 project presentations
This comprised introductory, half-day presentation sessions led by GEMS 1, 3 and 4 respectively at the very beginning of the in-country fieldwork. Held in Abuja, these sessions provided the opportunity for more detailed insight into, and discussion on, project management approaches, intervention strategies, partners and activities, progress to-date, and results and achievements. These opening sessions were also
5 Value chain analysis is a widely used tool in private sector development projects – to analyse the different stages in adding
value to a product or service. The role of value chains and their development is also central to the use of an M4P approach. Whilst M4P tends to look at the larger market system, it still recognises the value chain at its core.
18
useful for reviewing plans and priorities for meetings and site visits, lesson learning, and subsequent feedback and wrap up sessions.
3.2.3 Field Work & Stakeholder Engagement
The field work and stakeholder engagement during this MTR and Lesson Learning
Review was conducted by a team of 8 people, including 3 partners from the
Government of Nigeria. A member of the review team was partnered with a
government person, and each team responsible for in depth analysis of one of the
GEMS projects6.
Consistent with the Cluster Evaluation Approach, each of the 3 teams of evaluators
visited project intervention sites and met with stakeholders. In addition to the
evaluators, the VFM and M&E Experts also visited locations together with project staff.
The entire review team met on several occasions to review findings, including for the
project presentations from GEMS 1, 3 and 4.
A range of tools were used to gather information during the field mission stage of the
review. These included one on one meetings, as well as Focus Group Discussions.
Interviews were held with state level government partners, private and public sector
actors and individual beneficiaries, both direct and indirect. Across the reviews of
GEMS 1, 3 and 4 as part of the MTR, more than 70 separate meetings were held with
stakeholders.
3.2.4 Value for Money & M&E Analysis
Each of the GEMS projects was also subject to a Value for Money appraisal as part of
the Mid-Term Review. This was conducted using standard approaches of looking at
economy, efficiency and effectiveness.
The information gathered and reported to DFID in the Mid-Term Reviews of all 4
GEMS projects is used to contribute to the Lessons Learning Review.
3.2.5 GEMS 1, 3 and 4 lesson learning workshops
The Review team also used workshops to verify findings from the MTR and to gather
lessons learned from the GEMS management and field staff. Individual workshops
were held with GEMS 1, 3 and 4.
Originally conceived and scheduled to take part at the beginning of the Review period,
it was decided (when finalizing the Approach Paper) that these workshops would be
much more useful coming toward the end. The original proposed generic
outline/itinerary for the lesson learning workshops was reviewed toward the end of
fieldwork and the session questions were tailored to provide deeper insight into each
GEMS’ programming approaches in relation to what had been seen and heard during
project site visits, and partner and stakeholder consultations. The Reviewers ensured
6 Note that GEMS 2 was assessed under a separate review activity in 2013
19
learning objectives for the sessions and workshop overall were consistent with and fed
in to the Review design framework and priority areas for analysis in order to facilitate
analysis across the GEMS projects. Annex 6 contains the outline that was developed
for each of the GEMS 1, 3, and 4 lesson learning workshops. These outlines were also
useful for the Review team’s note taking and post-workshop analysis. Simplified
PowerPoint presentation formats of the same were used to guide discussion during
the actual workshops.
3.2.6 GEMS 2 Lesson Learning Review
Another important information source for the Lesson Learning Review was the GEMS
2 Lesson Learning Review. The GEMS 2 project was anticipated to run from early
2010 until 2015. DFID Nigeria took the decision to close GEMS 2 early based on their
assessment that it was unlikely to achieve its objectives within the remaining project
lifetime.
The 2013 Annual Review emphasized that the partners and stakeholders involved in
GEMS 2 (including DFID Nigeria, the World Bank, Coffey International, and the
Government of Nigeria) would benefit greatly from the learning that the project
experience had to offer at the level of partner experiences, and in the broader context
of market systems development projects that adopt an M4P methodology. It is on this
basis that DFID Nigeria commissioned a constructive review of the project. That
Review took place from 2nd to 6th December 2013. The final report presented the
findings and lessons learned from the Review process, as well as prioritized
recommendations for DFID and its service providers. This Lesson Learning Review
incorporates the findings and lessons learned from the GEMS 2 Review.
3.2.7 Limitations of the Data
With any review, the team involved will rely heavily on data and information that is
provided to them by the individual projects. However, the approach taken by the
Review Team in this assignment aimed to test the data in these reports by undertaking
several activities. The workshops with each team were an opportunity to discuss
results in more detail with individual staff. This was supported by extensive field visits
with beneficiaries, including meetings in which GEMS staff were not always present.
Independent verification of all information was not possible, but key findings were
discussed with Government of Nigeria representatives and other actors independent
of the projects.
The Review Team was also given access staff responsible for developing the reports,
and were able to question and test the basis on which the data was developed.
Overall, the Review Team was confident that the data used in this review was
adequate for the purpose.
20
3.3 Review Team’s Approach
Certain important factors have influenced the Review team’s lens and approach for
both the MTR and the Lesson Learning Review.
Firstly, the Review team took into account and recognised the different stages of
implementation of GEMS 1, 3 and 4, and what could realistically be expected in terms
of data collection, analysis and learning across each of the respective projects.
Whereas, GEMS 1 and 3 are effectively in their final implementation phase, GEMS 4
is in its initial implementation phase. Given where they are in their project life cycles,
GEMS 1 and 3 should have a very firm understanding of market dynamics, players’
interests, capacities and willingness to change, and clear vision for the project, which
they have successfully translated into a set of prioritized strategies that are yielding
results. On the other hand, GEMS 4 is likely to be still testing its strategies and is in
the early stages of understanding players and partners, whilst working to effectively
position itself and its interventions in order to maximize results and outcomes.
A second important consideration was the fact that GEMS 3 is not technically an M4P
project. GEMS 3 has developed and applied its Business Environment Improvement
Framework (BEIF)7 as its overarching approach across the project. For the most part
GEMS 3, unlike the other GEMS, is not working directly with enterprises – though all
of its interventions are intended to make doing business easier for them. In the case
of GEMS 3, therefore, the focus for lesson learning was not M4P in practice, but how
the project is able to contribute to M4P learning and broader market system
development programming.
In any review process, the potential exists for differences of opinion between either
members of the review team, or between the review team and projects themselves.
As each review team was responsible for in-depth assessment of one GEMS project,
regular de-briefing sessions were held by the reviewers to share findings and discuss
activities and issues from the field work.
The review team also was aware of some tension between GEMS projects, and
ensured that although open discussions were held with each project, no direct
comparisons were made between the performances of the projects. This reflected also
that each project was operating in different sectors, as well as at different stages of
implementation. It was clearly discussed with each GEMS project that the Lessons
Learning Review in particular was not about comparing projects, but rather learning at
the overall programme level.
7 More information on the BEIF approach and its development can be found at the GEMS 3 website: http://gemsnigeria.com/gems-
3/
21
4 Findings and Lessons Learned
At the core of all the GEMS projects is the goal of systemic change. Typically, M4P
programmes and market system development programmes more broadly are
conceptualised, designed, managed and delivered to catalyse a change process that
opens up new or existing market and/or business opportunities, which have tangible
benefits for the poor. By its very nature the M4P approach requires projects to
continually deepen their understanding and responsiveness to market systems.
Ultimately, project achievements and successes evolve out of the everyday realities
and choices of the players participating in target markets. Much of what happens within
markets is beyond the projects’ control. The lessons learned presented in this report
are intended to expand on the experience of implementing the M4P methodology,
which is core to the GEMS projects’ theories of change and intervention approaches.
Learning themes span project relevance, effectiveness, efficiency, impact and
sustainability and explore issues such as:
the appropriateness of the M4P approach in the different GEMS contexts and
the practicalities of applying M4P principles and methodology in complex and
dynamic markets;
the benefits of the contribution of an M4P approach to different sectors and the
underlying reasons;
lessons on the practice of engagement with different sectors, and how particular
GEMS components are affecting systemic change in markets.
Sections 4.1 through Section 4.6 cover the findings from the Review under the 6
thematic areas of the Review Matrix (refer to Annex 4). For ease of reading, each
section is structured the same, with an Introduction including the questions raised in
the Review matrix, a section on the Lessons and Findings and finally some
Conclusions and Recommendations for DFID and its service partners.
4.1 Formulation of an Appropriate Strategic Framework
4.1.1 Introduction
The findings and lessons presented in this section focus on answering the following learning questions:
What are the lessons learned in designing the strategic framework for an M4P
project for achieving pro-poor systemic change?
What could DFID and/or its service providers have done differently in terms of
translating the theory of change into an effective delivery strategy?
What are the lessons learned around adjusting the strategic framework to changes
in the political and economic context?
Aspects explored in answering these questions include: vision for a project and the
sector it will target, conceptualization, and effective positioning and engagement within
22
the sector. Intervention design and partner selection are also closely linked with
formulation of a robust and appropriate framework for delivery.
4.1.2 Lessons & Findings
The process of formulating an appropriate strategic framework (which in turn is the
foundation for a robust implementation strategy) should stem from a clear vision for
the sector being targeted, and requires good consensus across the main decision-
makers on the overarching rationale for the project and the direction it should take.
The GEMS 2 Lesson Learning Review identified several contextual factors in the
earliest project design stages, which contributed to GEMS 2 not achieving necessary
and clear strategic direction. This ultimately hindered the development of a robust and
well-focused framework for delivery. Key factors that stood out in this regard included
a significant degree of ambiguity in the original TOR, a lack of M4P expertise in the
formative stages of the project – both in DFID and in GEMS 2 project management,
and a resulting unsteady transition from original cluster competitiveness thinking to an
M4P approach8.
In fact none of the GEMS projects had a smooth journey from conceptualization
through to formulation of an appropriate strategic framework. GEMS 1 and 2 were
originally conceived as cluster competitiveness projects, and the decision to adopt an
M4P approach only came later as a result of their respective Inception Reviews.
8 The first two points relating to GEMS 2 ambiguity in the TOR and lack of M4P expertise on the part of both DFID and GEMS 2 are covered extensively in the GEMS 2 Lesson Learning Review Report and so are not revisited in this cross GEMS lesson learning review.
Lessons: Strategic Frameworks
Both DFID and the service provider need to be clear at the outset on the
agreed vision for a target sector, and consequently the purpose of the project
and/or interventions
As a follow on, the developmental approach the project will adopt needs to
be clearly communicated to avoid confusion or misinterpretation around project
focus and priorities
A lack of clarity or agreement on targets or approach will inevitably lead to
delays in implementation and difficulty in achieving targets
Conscientious efforts and expertise are required to position the project well
in the early stages, as this is vital to attract the right partners and build credibility
in the project; credibility starts from using experienced people in the project
23
GEMS 3 was also encouraged by DFID to adopt an M4P lens until such time as, in full
agreement with DFID, it moved firmly away from an M4P approach.
The early stage lack of clarity and weak focus and the repercussions of the re-direction
to the M4P approach (or in the case of GEMS 3, moving away from the approach)
should not be under-estimated. GEMS 2 made considerable headway during inception
in designing a strategic framework around a cluster competitiveness approach. It
ultimately lost some 13 months of effort establishing an implementation framework
(including sector engagement mechanisms, partner selection, and staff recruitment
decisions), which it then reformulated when advised to adopt the M4P approach.
In the case of GEMS 3, the turning point came in September 2012, effectively two
years after inception, when it conceived and adopted its Business Environment
Improvement Framework (BEIF)9 as its overarching approach for the project.
Whilst GEMS 1 relates to having a disruptive start to formulating its strategic
framework, it made a relatively smoother transition to an M4P approach, and even
confirmed that much of its market analyses using a cluster competitiveness lens
proved extremely critical in deciding which sub sectors of the meat and leather industry
it should focus on. This not only demonstrates that there are areas of overlap across
different market development approaches, but also that projects will blend different
approaches and tools as need be to achieve their aims. The GEMS projects’
experiences in this regard are explored further in the following section on Tools and
Approaches.
The GEMS projects all applied similar criteria for prioritizing and focusing their
interventions, including: relevance to the poor (potential for equity and inclusiveness,
scalability and replicability. The projects also pursued context specific approaches and
strategies for positioning the project and engaging with stakeholders. However, the
experience of each project in establishing early credibility in the eyes of industry
stakeholders and effectively engaging them in the research and design stages was
different.
Credibility starts from using experienced people in the project, who are then the
foundation for building ‘project’ credibility. This does not need to be project staff
exclusively, but can be service providers engaged by the project. However, there is a
big difference between ‘project credibility’ and ‘staff credibility’ – and the staff has to
come first.
GEMS 1 was able to gain credibility in the eyes of key stakeholders through provision
of sound technical and business information (based on using experienced and senior
local staff), relating to changing market opportunities and business drivers in the meat
and leather sectors. This strategy was enhanced through GEMS 1 initial cluster
9 BEIF is an approach developed by GEMS 3 for engagement with public and private actors. More information on this can be found in GEMS 3 Mid Term Review report.
24
development approach. As part of this, GEMS 1 established two stakeholder working
groups (SWGs) during the early part of the inception phase, and the membership of
these SWGs was largely drawn from the private sector. These groups proved to be
useful sources of information and acted as points of reference while GEMS 1 carried
out its overall review of the meat and leather sectors. During the process of developing
and refining its strategic vision for the two separate industries, key members of the
sector and the SWGs themselves were used to test suggestions and some of the
proposed strategies.
Even though after the initial Inception Review GEMS 1 was asked to realign to a more
focused M4P approach, the information and stakeholder credibility built up under the
cluster approach proved invaluable, and assisted in positively influencing both public
and private sector stakeholders and in identifying appropriate partners. This further
emphasises the need for strong understanding of the market system and drivers for
change, allowing flexibility in implementation strategies.
The experience of GEMS 2 was that the project had a difficult task to carve out its own
niche as a market development programme, and to make itself understood in the
market place. In hindsight, it may not have been prudent for GEMS 2 to drop the
original engagement mechanism and partners planned under the cluster development
approach as these may have worked just as well under the M4P approach. Initially the
GEMS 2 TOR had conceptualized the development of a Construction Industry
Development Board (CIDB), as a platform for stakeholder engagement in bringing
structure and improved competitiveness to the industry. GEMS 2 put considerable
efforts into sector consultations in preparation for forming the Board, and there was
significant interest from private sector partners looking to join. In adopting the M4P
approach, GEMS 2 (and - according to Coffey - DFID also) opted to abandon the
concept of the CIDB. The concept and the composition of the network could have been
retained or re-formulated to facilitate a two-way dialogue and feedback on project
direction. Instead this valuable platform for industry engagement was lost, as was the
momentum and energy initially created around it.
GEMS 3’s strategy has been to approach and initiate dialogue through public-private
engagement mechanisms (PPEM) with high-level stakeholders and public-private
dialogue (PPD) with grassroots level stakeholders to introduce its product offerings of
tax, land and investment reform, and explain the basis for partnership and way
forward. GEMS 3 has taken a very pragmatic approach to engaging with states and
local government areas (LGAs) – effectively shopping around with its product
offerings. Whilst this has resulted in a non-uniform mix of work streams across states
and LGAs, it has proven to be a sound strategy given the political economic context
and depending on state and local government willingness to embrace reforms in tax
and land especially. More on GEMS 3’s actual approach for engaging and forming
partnerships is explored in Section 4.2 Tools and Approaches.
25
GEMS 4 was able to achieve a high level of visibility within the first eight months of its
existence, and this was helpful in promoting market consolidation, and widespread
programme acceptance and recognition. It has been successful in establishing
credibility and influencing players especially in the private sector through joint process
with them. This is demonstrated through GEMS 4’s success in bringing a good cross
section of market leaders to the table, including the Fresh Fruit and Vegetable Dealers
Association of Nigeria (FFVDAN).
4.1.3 Recommendations on Strategic Framework
The process of applying the M4P approach from the outset will help projects ‘get it
right’ with respect to putting core programming elements in place. That process begins
with having a clear vision for the sector and for target beneficiaries in that sector; this
in turn requires clear understanding between DFID and the service provider on the
overarching rationale for the project (including its positioning in regard to other projects
and programmes) and the direction it should take, as well as sound communication
strategies for taking the project out to the market place.
What was in evidence from all 4 GEMS projects was the importance of a strong
knowledge of the target sector. In particular, this strong foundation of information (and
credibility) allowed GEMS 1 and 3 to adjust implementation strategies based on the
need of the donor or stakeholders.
Common factors for project success include establishing early credibility in the eyes
of institutional/industry stakeholders and effectively bringing them into the research
and design stages. Building credibility starts with the project staff and their ability to
clearly communicate the purpose of project, its benefits and the planned approach,
and to then effectively translate this into a strategic framework for delivery.
4.2 Tools and Approaches
4.2.1 Introduction
The section explores the GEMS project experiences in developing and using tools and
approaches for market system development programming more broadly. The findings
and lessons focus on answering questions on intervention design, partner selection
and delivery approaches, all of which are, in fact, also integral to formulating an
appropriate strategic framework for delivery.
Specifically, the questions included in the review framework are:
What, if any, are the trade-offs programmes have made in endeavouring to
implement the ideal M4P approach in a practical context?
What has worked and what has been learned with respect to facilitation
approach?
What has worked and what has been learned with respect to partner selection?
26
Of the various interventions, what approaches have proved most effective? Are
there commonalities across most successful / least successful?
Has the sequencing of tools and approaches worked effectively?
4.2.2 Lessons and Findings
The lessons and findings under Tools and Approaches are divided between 3 learning
areas: Intervention Design and Partner Selection, Delivery and Theory of Change.
A. Intervention Design and Partner Selection
One of the greatest benefits of M4P programming is that the actual process of doing
a thorough market analysis using an M4P lens should lead to robust intervention
design. Such analysis will help a project find the right interventions and the right people
to work with.
An example of an appropriate strategy for intervention design and partner selection
can be seen in how GEMS 1 chose to work with the feed manufacturers to introduce
improved ruminant feed for small scale farmers. Initially, the project after identifying
and contacting seven existing feed companies, selected one feed company to pilot the
Lessons – Tools & Approaches: Intervention Design & Partner Selection
Intervention design is achieved through strong stakeholder analysis and
deep understanding of all the market players (public and private), their current
and potential future roles in the market place, and realistic perspectives on
their interests and motives, which will steer partner selection
Market analysis needs to be front-loaded to ensure project strategy and
early stage interventions are well grounded.
Analysis needs to be sufficiently well developed so as to support the
design of interventions that have a clear rationale for market adoption and
uptake
The design and application of effective mechanisms and tools for
stakeholder engagement ultimately contribute to sustainable outcomes.
In developing its intervention portfolio, a project needs to have a mix of
intervention strategies and partners to drive ‘quick wins’ for direct impact on
the poor, balanced with a selection of what are likely to be longer, slow-burn
initiatives (e.g. that involve policy or institutional change).
Project’s should recognise the risks related to relying on a single partner
across interventions or at the core of an intervention area; initially, a project
may have no choice but to work with one partner, but the project’s intervention
strategies should include opening up opportunities for more players in due
course.
27
intervention and manufacture the first batch of small ruminant feed. Later the six other
companies were also re-approached as the project recognized it would be considered
a higher risk intervention, and all partners were not likely to succeed. Currently three
companies are active in the market place, whilst others are being assisted to come up
to speed in what is now a growing market. These developments in the market place
are leading to improved products and services for small scale farmers.
GEMS 4 has been faced with different market dynamics in its intervention areas. When
developing its mobile money intervention plans, GEMS 4 was faced with the challenge
of there being too many ‘takers’ interested to partner with the project. These were both
mobile money operators and super agents in the mobile money sub sector (the
rationale being that the project will facilitate the recruitment and training of smaller
scale agents to operate along trading corridors). The project did not know which the
best choices would be. GEMS 4’s solution was to start with multiple partners as part
of the learning process.
GEMS 3’s approach for partner engagement and selection is particularly noteworthy.
The 2014 MTR highlighted the effectiveness of GEMS 3 Public Private Engagement
Mechanism (PPEM) and Public Private Dialogue (PPD) mechanism for entry into new
states and Local Government Areas (LGAs) and to gain further traction in its existing
states, LGAs and partnerships. GEMS 3 uses the engagement (PPEM) and dialogue
mechanisms (PPD) as appropriate at federal, state and grassroots levels to not only
promote its ‘product’ offerings and solicit and measure buy-in, but also for strategy
development, solution finding, feedback loops, and verification. In many instances the
PPD has represented the first time public and private sector have come together to
address specific agendas and jointly resolve issues on tax, land and investment in the
context of business environment improvement. GEMS 3 PPEM and PPD events are
very strategic; they help build and reinforce staff and project credibility, events are well
publicized, there is careful choice of participants and high levels of inclusiveness, and
the project openly drives partners to make decisions and follow up with action.
A core idea of M4P is to ensure existing actors are not excluded by projects trying to
‘pick winners’. Typical to the M4P approach, the GEMS projects (as facilitators) have
sought to work with multiple partners (in both public and private sector), and with less
focus on any single specific partner or organization. This offers greater flexibility, but
it also implies the need for practical tools and approaches to support partner
assessment to avoid the risk of being stuck with a non-performing or inappropriate
partner.
One of the weaknesses of GEMS 2 intervention design and partnering strategies was
that the project had only one or two implementation partners across its various
intervention areas. Key partners such as Lagos State Vocational Education Board
(LASTVEB) and Lagos Waste Management Authority (LAWMA), because of their
complex bureaucracies, took much longer than anticipated to deliver desired
outcomes. At the same time, the project had primarily partnered with a portfolio of
28
weak and/or nascent associations (e.g. the League of Construction Engineers, the
Nigerian Institute of Builders, Association of Building Artisans of Nigeria, etc.). These
government bodies and industry association representatives all gave very positive
feedback during interviews with the Lessons Learned Review team. They had clearly
valued the technical support provided by GEMS 2, but their weak capacity and the fact
that most had to be carried each step of the way by the project raised doubts over the
choice and range of partners and the likelihood of scalable and sustainable impacts in
choosing to stick with those partners.
B. Delivery
The notion of ‘best fit versus best practice’ in applying the M4P approach came up
frequently in discussion with GEMS 1 and 4 during the MTR process and in the lesson
learning workshops. GEMS 1 and 4 use the M4P approach as their main guide for
how they engage with market players. These projects have also seen the need to
apply value chain development (GEMS 1), social marketing (GEMS 4) and action
research (GEMS 1 and 4), and to opt for direct intervention support to stimulate
markets, test a ‘theory’ or to prove a business case. The GEMS projects consider
these tools and approaches intrinsic to the M4P approach rather than somehow
outside of ‘best practice’ M4P methodology.
GEMS 4 saw the need for direct intervention support to trial the use of plastic crates
for bulking and transporting fresh tomatoes; the rationale being that the key
collaborating partners, mostly poorer small scale producers and marketers, needed to
be convinced on the commercial gains accruable by switching from traditional woven
Lessons – Tools & Approaches: Delivery
Implementing an M4P programme requires that from inception onward applied
approaches need to be analytical, entrepreneurial and continually responsive to
market system dynamics.
A market system development programme will invariably use a mix of tools
and approaches when applying an M4P methodology.
Action research is a key tool in M4P programme management for attempting
to bring new partners together, prove a business case, and for deepening
understanding and testing the market as to what can work. It should also contribute
to projects establishing credibility in their target sectors.
Adjusting interventions to account for market distortions, rather than trying to
‘fix’ the distortion is the preferred approach.
Systemic change takes time and partnerships and interventions will inevitably
evolve with varied degrees of success.
29
baskets to using the plastic crates. GEMS 4 opted for a full-blown demonstration using
the crates to transport producers’ tomatoes from supply points in the North to regular
market outlets in the South. GEMS 4 was very hands on in the trial and it involved
additional work to incentivize and engage partners, but this is a good example of an
M4P project using action research as part of the market diagnostic process, and as a
means for ‘testing’ an opportunity in the market place.
As mentioned earlier, the M4P lens if applied well will not only facilitate market
diagnostics, but will also point the way to intervention options. Applying an M4P lens
should also sharpen a project’s ability to continually ‘read’ the market and intervene
accordingly. This process is multi-dimensional and requires people and applied
approaches that are analytical, entrepreneurial and continually responsive to market
system dynamics. For example, GEMS 1 has been able to stimulate copying and
crowding in10 under several activities around feed finishing and red meat processing.
This was based on good market analyses and deep understanding of the sub sectors,
which led to well-developed intervention plans. Of note was the project’s direct
intervention approach to prove the business case for making ruminant feed. The
project had identified seven companies that were producing and distributing poultry
feed, but at the time none of them were producing small ruminant feed. GEMS 1 ran
a pilot with one of the identified companies, which was provided with comprehensive
technical support and assisted to produce ruminant feed that the project then
purchased from it. The ruminant feed was distributed to farmers, and the resulting
improvement to animals led to farmers demanding the product. Based on this success
and increasing market demand, at least three of the feed companies are now
(independently of the project) investing in increased production, developing new
products targeting sub-categories of feed finishing, and undertaking marketing and
promotion activities. Very early on, GEMS 1 reduced direct support to the pilot, and
instead focused on finding ways of deepening and scaling the business opportunity.
GEMS 1 did this by engaging government employed paravets to become suppliers of
the improved feed in addition to their role in providing animal health services to
farmers. A successful model is evolving, which focuses on existing actors and
improving the services they are able to deliver.
The work GEMS 1 is doing with the feed manufacturer, Animal Care in linking it with
small scale producers and the large multinational meat company, Master Meat, which
in turn supplies the South African supermarket retailer, is very much a value chain
development approach. The intervention approach is highlighting the improvements
and benefits that can be achieved by properly feed finishing animals, as well as
providing scale to the sector by enabling feed manufacturers to profitably produce
10 This is an M4P term for when interventions catalyse copying or bring other players and functions into the market system so that it works better for the poor. Crowding-in can result in enhanced breadth (more transactions in the core of a market), depth (supporting functions) or reach (new areas or markets).
30
feed, and linking increased numbers of small scale producers to growing ‘high end’
retail market outlets via a lead supplier. The challenge for GEMS 1 is to ensure that
the strong actors involved do not, by way of their success, exclude the poor from the
value chain.
GEMS 4 applied its M4P lens differently again in order to achieve momentum early on
in its interventions in micro distribution and wholesale buying groups (WBGs). GEMS
4 was able to get earlier buy-in of women into WBGs by working in collaboration with
organisations that were already well established locally and had already developed
the high levels of trust required to work closely with women in community. The project
saw the opportunity - and the need - to successfully position itself and introduce its
planned interventions by identifying and working with existing WBGs rather than trying
to form these groups from scratch, which would have been more time consuming and
more challenging.
GEMS 1 has intervention strategies focused on influencing public sector’s role across
the meat and leather sectors. For example, GEMS 1 deliberately supported the
development of and worked through industry stakeholder groups to influence public
sector. This has been fundamental in establishing with the public sector that the
leather value chain is an important component of the livestock sector worthy of
support. GEMS 1 has also embedded itself in the Federal Ministry of Agriculture and
Rural Development (FMARD) Agricultural Transformation Agenda (ATA). The
project’s presence on the ATA has been pivotal in introducing government budgetary
allocation for transformation activities in both meat and leather sectors.
GEMS 3 also opted to embed its project experts in key line ministries to help drive
reform agendas in land, tax and investment. This means GEMS 3 has been in a better
position to support capacity building, and has been able to foster more effective
collaboration, and in some cases more immediate problem solving. GEMS 3 also
considers its strategy of embedding expertise as way to also embed and
institutionalize reforms, and make them more sustainable.
These strategies may not reflect the textbook ‘facilitative’ approach we may associate
with the M4P methodology. The projects have gone this route in order to influence,
fast-track and where possible lock down policy changes and reform, which otherwise
may not have been possible in the lifetime of the projects. However, just as previously
mentioned, the main factor underpinning the projects’ success in this approach has
been the people - and their ability to quickly build credibility, and establish the value
add in having a project work more ‘hands on’ in this way. Of course, being able to build
harmonious relationships and provide demonstrable results in a reasonable timeframe
is also very important to success.
The GEMS projects have also encountered circumstances whereby a constraining
policy is unable to be changed (or at least change is unlike to bring positive results for
the sector within a reasonable timeframe). The GEMS 1 experience with the
government subsidy of the Export Expansion Grant (EEG) offers a good example of
31
recognising that it is not always feasible to try and ‘fix’ the market distortion, but instead
it is better to work to resolve problems independently of the issue.
The EEG provides exporters with a rebate of 30% of the value of the leather they
export. The EEG is not linked to quality, and therefore tends to drive volume of skins
into the export market. Local markets for finished leather goods suffer as a result as
local leather is difficult to procure. Adding further to the challenges in the leather value
chain is the use of cattle skins for food; a local product called ‘pomo’. For the seller of
the animal there is greater value in cattle skins for food compared with leather.
GEMS 1 approached this sector looking at ways to work within these constraints.
Whilst they are also working at the government level to try and reform the EEG, they
have not waited for this to be resolved. Their approach has been to instead look for
other ways to provide pro-poor benefits from engaging in the sector. Interventions
around improving the conditions for tannery workers, techniques to improve
processing of skins and also in leather production to increase value have all been
implemented. Good results are being seen in some of these activities.
Similarly, GEMS 4 is hopeful it can influence change in the Central Bank of Nigeria
(CBN) policy on Know Your Customer (KYC) requirements, which currently restricts
mobile money payments of informal micro and small enterprises. However, it is not
basing its success on this policy change taking place. Whilst GEMS 4 continues to
engage CBN on reform options, it already envisages a ‘plan B’, which would focus on
using third party agent banking models to facilitate cash flow to and from rural areas.
This initiative could move more quickly as CBN is interested to explore agent banking
as part of its broader strategies for increasing rural financial inclusion.
32
C. Theory of Change
The previous section illustrated how the GEMS projects have applied a range of tools
and approaches to design and deliver contextually sound interventions. The important
thing to have clear is how a successful pilot, action research initiative or intervention
design will then be translated into meaningful and growing business opportunities (or
in the case of GEMS 3, meaningful and sustainable improvements to the business
enabling environment) that benefit target beneficiaries.
The design and application of the Theory of Change (TOC) and intervention logics
(also referred to as results chains) lie at the heart of M4P tools and approaches for
ensuring relevance, effectiveness, impact and sustainability across project
interventions. Use of intervention logics is an essential component of the Donor
Committee for Enterprise Development (DCED)11 Standard for measuring and
evaluating achievements of private sector development programmes. This section of
the report examines how the GEMS projects have developed and applied the theory
of change and intervention logics to strengthen project management, and talks to the
benefits of the system as well as challenges the projects have faced.
The section links back to questions asked in the Evaluation framework such as what
has been learned about project management, oversight, feedback loops and effective
M&E systems?; possible trade-offs programmes have made in endeavouring to
implement the ‘ideal’ M4P approach in a practical context?; and what could
DFID/service providers have done differently in terms of translating the theory of
change into an effective delivery strategy?
11 More detailed information on the DCED standards and how they are applied to results measurement can be found at http://www.enterprise-development.org/page/measuring-results
Lessons – Tools & Approaches: Theory of Change
A clearly defined intervention logic (or business case) helps keep activities
focused on the target beneficiaries, and should be robust enough to guide
intervention decision-making, including critical and timely decision-taking for
dropping an intervention or partnership. It also provides a path to an exit
strategy.
Developing robust intervention requires the right lens, tools and
approaches for monitoring and evaluation to identify and address weaknesses
in intervention strategies as outcomes unfold beyond the direct influence of the
project.
It requires a well-functioning intervention logic system for the full benefits
of the approach to be realized; the continuous management cycle of designing,
re-examining and revising intervention logics based on unfolding results draws
significantly on project resources.
33
GEMS 1 undertook and “passed” a mock DCED audit12. In this regard it is highly
advanced in applying DCED principles and practices and offers valuable observations
and lessons on practical application over the life of a project. In terms of system, every
active intervention / intervention area under GEMS 1 has a fully up-to-date intervention
logic. At the same time, all interventions that have been closed off by GEMS 1 over its
lifecycle have been documented (via the intervention logic) with respect to reason(s)
why the market intervention had failed; this may be due to failure to understand a
market constraint, or where traction of the intervention failed because of unforeseen
problems, or due to inappropriate assumptions and/or envisaged linkages, which
never materialised.
The GEMS 1 project team meets twice a year to conduct a technical review at which
time each intervention manager has to present against their broad intervention logic
covering their intervention/output area. Intervention managers are required to explain
any changes in direction and strategy and these are discussed based on reference to
the logic.
A continuous cycle of re-examining and revising intervention logics based on unfolding
results is time consuming, but results are worth this time investment. An example is
GEMS 1’s interventions in feed finishing; intervention logics for this project component
have undergone substantial changes and a number of iterations as the programme
has evolved and changed.
In the case of GEMS 4, the TOC is a combination of intervention logics that feed into
or ‘explain’ a broader overall logical framework analysis. GEMS 4 is using its overall
TOC to present a picture of the multiple interventions that it is using or will use to reach
the project’s end goals. Once an outcome has been identified, the next step is to
employ logic models to explain how that outcome will be produced. Thus, the TOC is
necessary for summarizing work at the strategic level, while logic models are used to
illustrate the tactical or intervention level links of the envisaged change process.
Although it is early days to observe or learn from the GEMS 4 model working in
practice, the GEMS 4 approach illustrates the usefulness of linking one overall
programme TOC to a number of logic models.
GEMS 1 uses a step by step process to reduce the workload on intervention managers
and improve the quality of analysis across its TOC and intervention logics. Initially, a
brief concept note format is used to write up the business case for an intervention; this
note is the basis for closer scrutiny and team discussion; it is then refined, and used
to produce a simple intervention logic for early stage implementation; regular reviews
and refinement of the intervention and logical model continue thereafter.
12 Information on what is entailed in such a mock audit can be found athttp://www.enterprise-development.org/page/audits
34
GEMS 1 experiences highlight the levels of expertise required not only in developing
robust intervention logics, but also managing them as project implementation gathers
pace. Expertise is also required at field level to translate back and forth between
intervention logics and the realities of results in the market. GEMS 1 had to invest
heavily in extensive coaching to develop intervention managers ability to ‘argue’ the
progress of their projects (and the constraints faced) on the basis of their documented
intervention logics.
As with many M4P programmes, the GEMS projects are complex and ambitious and
as described above it takes significant resources to continually develop, revisit, test
and refine intervention logics across the range of interventions a typical M4P project
will be managing at any given phase in its project lifetime.
The rationale for the intervention logic is to define the background information and
data collection that is needed to be confident that change is happening and to provide
robust evidence that systemic change is starting to happen and is developing along
the trajectory envisaged in the intervention logic. Without a well-functioning
intervention logic system, which is integrated into project management structures, it is
doubtful whether a project can really benefit from the advantages of the approach.
A weakness in GEMS 2 intervention strategies was that the project developed and
continued to focus on a portfolio of interventions, which in some cases were overly-
experimental, were based on speculative results chains and not grounded in robust
understanding of market dynamics and players’ interests. In some cases, these were
overly reliant on weak or nascent institutional partners. The failure to re-visit
interventions and partner choices in good time contributed to the slow pace and slow
delivery that ensued through the project’s lifetime. In this example, the intervention
logics should have been the guiding tool for knowing when to re-engineer or drop an
intervention area or partnership.
In contrast, the temptation to continue to intervene in a sector that is providing positive
results can be strong. The GEMS projects are assessed on the numbers (targets)
generated, and the value for money of results. Added to this is the need for positive
market system changes to be able to be sustained by market players after the project
intervention stops. Referring to the GEMS 1 successes in feed finishing described
earlier, in this case, the project is yet to move out of the feed finishing sub sector in
areas where crowding in and replication are taking place. In this example, the
intervention logics should be the guiding tool for knowing when to exit.
Findings from the GEMS 3 MTR point to another learning perspective with respect to
what a project needs to measure as you monitor and assess results along the theory
of change and intervention logics. GEMS 3 interventions are designed to improve the
business environment by reducing obstacles to private sector growth in the hope that
businesses will then change their behaviour, which in turn will lead to increased
investment, innovation and job creation for the poor. GEMS 3 theory of change is
based on waves of impact: interventions focus on addressing business environment
35
(first wave of impact), which then impacts private sector (second wave of impact) with
further impact on enterprise growth and the poor (third and fourth waves of impact).
GEMS 3 has developed a range of tools and approaches to measure results at these
different levels of impact, including satisfaction and viability indexes, which the project
uses to gauge early signs of sustainability of interventions. However, adequately
capturing changes in behaviour (on the part of ministries, departments and agencies
on the one hand and businesses on the other) is also key to understanding what
systemic change is taking place and if this is contributing to sustainability of reforms
beyond the life of GEMS 3.
The MTR of GEMS 3 pointed to the need for the project to ensure its tools and
approaches for monitoring adapt as outcomes from intervention strategies unfold
beyond the direct influence of GEMS 3. In market systems, projects must examine
changes taking place to identify, deepen understanding of, and counter possible
systemic weaknesses that are emerging as interventions unfold. This includes
unintended outcomes and/or behaviour changes that if not addressed could potentially
undermine positive systemic changes taking place. This is very much in line with the
M4P approach and is less to do with numbers and targets and much more to do with
changes in roles, new systems replacing old, displacement, changes in relationships
and power balances, behaviours, and incentives to change.
4.2.3 Recommendations on Tools & Approaches
In analysing the GEMS programme, it is clear that using a mix of tools and approaches
to research, analyse and design intervention plans is not at odds with an M4P
approach, but in fact one of its strengths.
What is important is the capacity for project management and field staff to continually
apply the M4P lens to shape market analyses, action research initiatives, partnership
choices and agreements, and what needs to be monitored and measured over time.
Applying an M4P lens should also sharpen a project’s ability to continually ‘read’ the
market (what is emerging in terms of changes in practices, behaviour, roles, attitudes,
etc.) and support accordingly. This process is multi-dimensional and requires people
and applied approaches that are analytical, entrepreneurial and continually responsive
to market system dynamics, and to those being impacted by those dynamics.
There are good sound examples of where the projects have applied a more direct
intervention approach to influence, fast-track and where possible lock down policy
changes and reform, which otherwise may not have been possible in the lifetime of
the projects. The main factor underpinning success has been the technical capacities
of management and staff - and their abilities to quickly build credibility, and establish
the value add in having a project work in this way. However, being able to build
harmonious relationships and provide demonstrable results in a reasonable timeframe
is very important to success.
36
The GEMS projects offer valuable learning on the use of the log frame, Theory of
Change and intervention logics as tools for project management. In order to service
as useful management decision-making tools, projects need to ensure that individually
and collectively these tools are kept ‘live’; they need to be well aligned to each other
and continually revisited and reviewed based on emerging and credible evidence from
the market place.
All the same, the continuous management cycle of re-examining and revising
intervention logics based on unfolding results is time consuming. Expertise is required
not only to develop robust intervention logics, but also manage them, and the system
overall as project implementation gathers pace. Ultimately without a well-functioning
intervention logic system, which is integrated into project management structures, it is
doubtful whether a project can really benefit from the advantages of the approach.
As GEMS 1 will maintain, whilst application of the DCED system requires a lot of work
and effort, it is deemed to be worth it. There is already global recognition that the
standards and audit processes can bring value to project management and evaluation
processes with respect to increasing credibility in results measurement and improving
effectiveness of delivery. There is scope for the DFID-funded GEMS projects to
contribute learning on the practical application of DCED tools and approaches to other
GEMS projects in the portfolio and also other M4P programmes in Nigeria.
4.3 Management and Oversight
4.3.1 Introduction
The findings and lessons presented on management and oversight build on the GEMS
projects’ experiences of applying M4P tools and approaches in practice by taking a
closer look at their experiences in designing and managing an appropriate M&E
framework for their projects. In line with the review framework, the section also
elaborates on observations and learning with regard to synergies and collaboration
across the GEMS projects and what has been learned about how to coordinate
effectively between M4P programmes both on the part of DFID and its service
providers.
4.3.2 Lessons & Findings on Management and Oversight
The lessons on Management and Oversight are divided into 3 categories, M&E
Frameworks, Use of Other Management Tools for Maximizing Impact and Synergies
and Collaboration across programmes.
37
A. M&E Frameworks
There is an overarching GEMS M&E framework in the form of the cross GEMS log
frame and GEMS Results Measurement Handbook. A critical finding from the GEMS
2 Lesson Learning Review was that the process of setting targets in the individual log
frame was compromised by having to align these targets to the cross GEMS log frame.
The result was that certain output targets developed by the GEMS 2 team were overly
ambitious (though not all), and were set as a stepping-stone to achieving overall
impact targets. In the end, output targets did not necessarily align with real or
perceived lead times for GEMS 2’s systemic approach to deliver results. This further
impacted on quality and usefulness of results chains, which were developed to reflect
intervention choices, but at the same time respond to overly ambitious indicators and
targets in the log frame. As a consequence these critical project management tools –
the log frame, TOC and intervention logics – were not well aligned to each other; nor
were they sufficiently ‘owned’ or used for project management decision-making by the
team on the ground.
The cross GEMS log frame and the individual GEMS project log frames focus almost
entirely on quantitative indicators. This has inevitably led to a culture across the
projects of results based management focused more on numbers and less on
qualitative changes in the market place. In the earliest stages of GEMS log frame
design, the assumption may have been that achieving the ambitious numerical targets
at output, outcome and impact level would be adequate reflection of underlying
systemic changes taking place, and that quantitative data collection and reporting
would be balanced with mechanisms for monitoring and recording qualitative results.
Lessons - Management and Oversight: M&E Frameworks
A combination of qualitative and quantitative indicators (designed with an
M4P lens) is required to ensure projects effectively capture systemic change.
The focus on meeting (contractually required) numerical targets can result
in positive changes to the market system not being adequately captured.
The use of ‘stories’ as part of the reporting is an effective way to
communicate results in combination with quantitative indicators, adding depth
to the numerical results
To be a useful tool in managing and delivering M4P programmes, the log
frames has to evolve with the project. Revisions have to make sense in the
context of M4P; staying consistent with the underlying approach.
The log frame, Theory of Change and intervention logics also need to be
kept aligned throughout the project lifecycle in order to serve as meaningful
project management tools at field level in particular.
38
As the projects focus primarily on achieving these numerical targets (as they are
contracted to do), the trade-off has been that project strategies, tools and approaches
for monitoring, evaluating, and reporting do not adequately address positive (and also
negative) systemic change across the GEMS target sectors.
This is not to say that the projects do not do self-assessment. The projects have taken
it upon themselves to track and report qualitatively, but approaches for measuring
qualitative results vary widely across the GEMS projects.
In its 2014 Annual Report, GEMS 1 uses personal comments or stories from
beneficiaries and other stakeholders to illustrate in more detail what the ‘numbers’
mean in qualitative terms under each of the project outputs. These are generally
placed as text boxes on the pages containing relevant discussion on a topic. This is a
useful and effective way to bring out more information on qualitative results and
deepen the understanding of the reader, and at the same time make the report more
readable. Such story telling could be used across more projects as a way of adding
meaning to reports that can otherwise feel quite remote from the project’s reality.
However, as a tool, case studies may not fulfil the need to demonstrate impact at
scale. Moreover, isolated “stories” are not the same as conducting systematic case
studies, which for M&E purposes would normally include interrelated qualitative and
quantitative elements. The use of stories in the GEMS 1 example is an effective
editorial device, but not a research method for evidence-based reporting on its own.
In addition to early stage impact assessments and case studies to gather and analyze
qualitative results, GEMS 3 intends to measure systemic change and sustainability of
its interventions in both the public and private sectors by the percentage of new or
improved products and services (services, regulations and reforms in the case of the
public sector), introduced through project facilitation, that are sustained in the market
12 months after project support has ended. Given that all GEMS 3 core interventions
are still in progress this cannot be measured and reported at this stage. As an interim
measure, GEMS 3 has developed and applies its ‘viability index’ as a litmus test to
gain feedback from key stakeholders on their perceptions on sustainability of
interventions. The index measures such variables as major challenges facing
organisations, operational viability and scale, financial and HR commitment, and
organisational planning.
Another important observation from the MTR and GEMS 2 Lesson Learning Review
is a lack of structure and consistency applied across projects in terms of reviewing and
revising log frame targets, and also intervention level outputs and targets. In the case
of GEMS 2, the project log frame did not evolve with the project with respect to timely
and structured review of outputs and targets, which meant that GEMS 2 increasingly
lacked ownership of the log frame. In addition, there was little ownership of targets
above intervention level on the part of GEMS 2 intervention managers. This growing
disconnect was not resolved and ultimately proved to be a core contributory factor to
GEMS 2 poor performance against its log frame targets. At the same time, GEMS 2
39
interventions in affordable housing work did not feature in the log frame outputs,
although the decision to work in this area would have contributed to outcome and
impact results had interventions advanced further. Lastly, the GEMS 2 team felt
aggrieved that they were not able to revise log frame targets to take into account what
they considered a “lost implementation year” in 2011
Overall, it appears that the GEMS log frames could be serving the projects better as
a management tool in several ways. As a critical project management tool, the log
frame has to evolve with the programme. Revisions have to make sense in the context
of M4P, namely that the underlying approach does not change even if the intervention
mix does. Revisions to targets should allow time for and balance with goals for deeper
and broader systemic change. The log frame must remain a useful tool for analysing
performance results over the life of a project; analysis of project performance is
significantly undermined when results achieved are clearly out of line with targets set;
there needs to be timely analysis of the reasons why this is the case and preferably a
documented process for appropriate corrective action. In this regard, the Annual
Review process could be more useful for more detailed analysis of the evolution and
status of the log frame indicators and targets to ensure these are well aligned with the
TOC, results chains and intervention strategies.
A key cross GEMS and individual GEMS project management tool is the GEMS
Results Measurement Handbook that was developed to guide practical application of
M&E across the GEMS projects using an M4P approach. The MTR findings confirmed
that the GEMS projects have made good progress in incorporating the tools and
approaches into their respective M&E frameworks, and have adopted and applied the
guiding methodologies for results measurement as per the handbook. What appears
to be missing is joint efforts by the GEMS projects to go back and reflect on the
usefulness of the handbook or to review and upgrade the approaches – looking, for
example, at if and what cost trade-offs there have been of using the three step
measurement approach and ways of improving effectiveness, and also revisiting
definitions and categorization since there are already examples of where revision
could be useful (e.g. the definition and measurement of full-time equivalent jobs in the
context of the GEMS projects). In addition to practical review and enhancement of the
handbook for future GEMS project usage, there is also an opportunity for the GEMS
projects to share more broadly on the benefits of applying the approach, and their
experiences on practical adaptation and application to maximize effectiveness in M4P
intervention design, management and delivery.
40
B. Use of Other Management Tools for Maximizing Impact
More often than not, conceptually and practically, the GEMS projects are trying to
successfully design and incentivize the uptake of new or improved models of doing
business or answer the question ‘what’s the business case?’ Arguably, as market
system development projects, the use of market models and business case
development to support analysis and decision-making would be practical tools to use
for project planning, intervention design and later when it comes to planning for
replication and scale up. Similar tools could be used to analyse post-intervention
results and/or to design ‘phase two’ strategies. For example, comprehensive economic
modelling and analysis could be commissioned to demonstrate economic benefits as
a result of GEMS 1 work in essentially creating a feed finishing market, and help
identify strategies for future growth and/or consolidation.
Another tool that appears missing from the GEMS programme design and delivery
components, but could be useful is the use of commercial style benchmarking. Applied
well, commercial benchmarking (either within a project and/or sector or across
broader, defined parameters) can be an incredibly powerful tool to drill down and
deepen understanding on business practices, why some businesses are performing
better than others, and to help identify priority areas for more support or emerging
good practices for wider dissemination. Such benchmarking is not only valuable to the
service provider, and project management teams, but it can also be enlightening and
helpful for target beneficiaries, and has knock on benefits for lenders, and others
servicing the supply chain. It is also well suited to M4P programmes where you often
have large numbers of small firms that are essentially trying to operate a similar
business model.
Lessons - Management & Oversight: Tools for Maximizing Impact
The complexities of measuring and managing M4P programmes justifies
the use of more innovative tools and approaches such economic modelling
and commercial benchmarking to deepen understanding of business
practices, business/sector performance, and opportunities for scaled impact.
41
C. Synergies and Collaboration across programmes
The lack of structured intra-GEMS and wider project dialogue and the very practical
challenges of coordination across the different GEMS components (and DFID-funded
State Level Programme more broadly) were noted in the GEMS 2 Lesson Learning
Review as was the need to appropriately and practically address the matter moving
forward.
Reflecting on GEMS 2 experience, analysis of the constraints to growth in affordable
housing highlighted the need for intervention in the following core areas – land
administration, financial markets, and public-private partnership (PPP) capacity;
Because technical expertise on each of these areas is concentrated in other projects,
GEMS 2 was reportedly told (by WB and DFID) not to develop interventions to address
identified bottlenecks on Land (GEMS 3), Housing Finance (EFINA), or PPP capacity
(NIAF). Instead GEMS 2 turned its focus to skills development and input supply
markets. While there was logic to this decision, the lack of effective coordination
mechanisms across programmes meant that strategies to address these other
important constraints to development of a pro-poor construction sector were not
prioritised and/or could not be sequenced in a way that would assist GEMS 2 efforts.
Given that the different GEMS components began at different times, are at very
different stages in their interventions, and experience shifting programming challenges
and priorities, it was always going to be difficult to successfully align cross–GEMS
interventions. In principle, it was hoped that synergies across the different specialist
GEMS components would be realised. In practice, efforts to build coherence across
the GEMS suite do not appear to go much deeper than the cross-GEMS log frame,
the M&E Results Measurement Handbook, and cross project support to establish
appropriate strategies for Women’s Economic Empowerment.
The fact that the discontinuation of GEMS 2 has not had any negative impact on the
other current GEMS components is also an indicator that there was very little cross
GEMS fertilization.
Lessons - Management and Oversight: Synergies and Collaboration
Development of frameworks for cross programme collaborations should
not be overly prescriptive, in particular given that M4P projects are likely to
evolve at very different paces
Such frameworks or agreements require clear input from funding bodies
(or neutral partners) in framing and managing collaboration or integration, as
well as roles and responsibilities for all partners
An appropriate funding mechanism that provides timely financial and
technical support will help bring about more collaboration.
42
The original design of GEMS envisaged a flexible funding facility, which each GEMS
could tap into to support innovations in the market place, including collaborative
ventures across the GEMS projects. The fact that only one GEMS project (GEMS 3)
ended up with a flexible facility is a point of contention with the other GEMS to the
point that the arrangement has been a barrier to collaboration between the GEMS
components.
The prospect of three new World Bank-funded GEMS entering the Nigerian economy
to work in ICT, Entertainment and Construction will inevitably increase the potential
for overlap in activities within the GEMS programme and across the broader suite of
SLPs. The same challenges of how and to what extent cross collaboration is practically
possible may well emerge.
It is not enough, as is the case currently, to simply collate and combine results in a
common GEMS log frame. DFID Nigeria and the WB need to review programme
coordination mechanisms and design realistic options and positive incentives to
interlink cross-GEMS and cross-DFID Nigeria growth programme portfolio. Some
straightforward and immediate actions could involve the convening of the different
M4P projects and SLPs to participate in learning events around best practice on
market system development programming – these would be particularly helpful for
field level staff; a comprehensive mapping exercise could help identify scope of
projects, sector activities and potential ‘hot spots’ for coordination and/or collaboration.
A key issue raised by DFID Nigeria in producing this report was the inevitable level of
competition and ‘ownership’ of beneficiaries, which comes about due to the way
projects are evaluated; accountability and attribution, for example, tend to make
projects nervous about collaboration at intervention level. A key driver behind
cooperation in the private sector is the prospect of both partners gaining and profiting
from the alliance. This may not be an appropriate reward when it comes to cross
GEMS cooperation. It is outside the scope of this study to explore more deeply
practical measures for how best to incentivize cross GEMS cooperation or deeper
integration of programmes. However, the overall level of DFID investment and tighter
budgetary controls on the one hand, and DFID’s vision for improved governance and
unleashing growth potential in Nigeria on the other represent a strong case for greater
cross GEMS integration.
4.3.3 Recommendations on Management and Oversight
The cross GEMS log frame and the individual GEMS project log frames focus almost
entirely on quantitative indicators. This has inevitably led to a culture of results based
management focused more on numbers and less on qualitative changes in the market
place – which is at the core of M4P. The trade-off has been that project monitoring,
evaluating, and reporting do not go far enough with respect to measuring positive (and
also negative) systemic change across the GEMS target sectors.
43
There is also a lack of structure and consistency applied across projects in terms of
timing and rationale for reviewing and revising log frames. In this regard, the Annual
Review could be more useful with respect to more detailed assessment of the
evolution and status of the log frame indicators and targets to ensure these are well
aligned with the TOC, results chains and intervention strategies.
Overall, it appears that the GEMS log frames could be serving the projects better as
a critical project management tool; the log frame has to evolve with the programme
and remain a useful tool for analysing performance results over the life of a project;
analysis of project performance is significantly undermined for example when targets
are clearly far out of line with results being achieved. It would also be useful to have a
documented process to support log frame revisions.
The GEMS projects could share with other M4P programmes on the benefits of
applying the approach, and their experiences on practical adaptation and application
to maximize effectiveness of intervention design, management and delivery.
The complexities of measuring and managing M4P programmes justifies the use of
more innovative tools and approaches such economic modelling and commercial
benchmarking to deepen understanding of business practices, business/sector
performance, and opportunities for scaled impact.
Synergizes and cross GEMS collaboration do not appear to go much deeper than the
cross-GEMS log frame, the M&E Results Measurement Handbook, and cross project
support to establish appropriate strategies for Women’s Economic Empowerment.
Given that the different GEMS components began at different times, are at very
different stages in their interventions, and the almost inevitable level of competition
and ‘ownership’ of beneficiaries, which comes about due to the need to report on
attribution, it was always going to be difficult to achieve closer alignment at intervention
level.
The original design of GEMS envisaged a flexible funding facility, which projects could
tap into to support innovations in the market place, including collaborative ventures
across the GEMS projects. Whilst in the GEMS projects this was not used for such
collaboration, the need for funding mechanism to encourage collaboration is important.
In retrospect, a structured and workable model for cross GEMS project collaboration
would need to have been devised during the inception of GEMS and have included
positive incentives to encourage partners to seek opportunities to work together.
With the three new WB-funded GEMS entering the Nigerian economy, it is timely for
DFID Nigeria and the WB to review programme coordination mechanisms and design
realistic options and positive incentives to interlink cross-GEMS and cross-DFID
Nigeria growth programme portfolio.
44
4.4 Value for Money
4.4.1 Introduction
The findings and lessons presented in this section focus on exploring what lessons
have been learned about implementing M4P programmes to maximize the potential to
increase impact and VFM. Under this broader question, the report looks at the
following aspects of VFM:
Learning on VFM as a concept, and its application in project implementation;
External benchmarking (for deeper analysis of economy, efficiency, and effectiveness);
Equity as a key aspect of VFM;
Lessons on overcoming challenges with respect to timescale and attribution in M4P programmes;
Good practice examples of VFM actions across the GEMS projects, which offer learning for other projects in Nigeria and beyond.
Lessons - Value for Money
VFM measures ideally need to be framed from the outset as a
management and measurement tool.
The ability to interpret the value of a project originates from the project’s
Theory of Change and intervention logics, both of which need to be revised
regularly.
In order to be meaningful, narratives need to be provided along with
comparisons on economy and efficiency indicators to explain particular
contextual differences or challenges.
Assessing VFM is a continuous, evolving process of comparison with cost
benchmarks and current practice and seeking/achieving improvements to
these. Even though comparison across a broad range of projects may
sometimes diminish its utility, external benchmarking research and analysis
can still be useful.
Achieving equity goals has cost implications as demonstrated by their pro-
poor and women reach results and associated costs.
Two major challenges for VFM in M4P programmes come from timescale
to achieve impact at scale, and attribution to show causality in a complex
operating environment.
A variety of methods need to be used together to address attribution with
regard to outcome and impact.
45
There has been a steep learning curve for GEMS projects (which were orchestrated
before the DFID requirement to feature VFM measures from the outset in the Business
Case), which is reflected in the VFM sections of both Annual Reviews and the projects’
own Annual Reports (which include VFM self-assessments). The recent ITAD-led
review13, as well as the MTR in May-June 2014 of each GEMS project both confirm
this, and further make suggestions to improve the coverage and overall VFM narrative.
The content of the GEMS projects VFM reporting, along with their VFM frameworks
are getting more sophisticated as learning is accumulated and applied back into
project management. This evolutionary learning has been significant.
Section 4.2 and 4.3 on Tools and Approaches and Management and Oversight
respectively have examined the GEMS projects’ experiences in developing and
applying the theory of change (TOC) and intervention logics. A project’s TOC is also
highly relevant to understanding and interpreting the value of a project. All the more
reason for projects to devise strong theories of change – and results chains - and re-
visit them often to check the evidence base and test the assumptions - both high-level
(firm growth, poverty reduction, etc.) and detailed (deadweight loss, displacement
effects, income or job multipliers, etc.). The more thorough the articulation of the theory
of change, the more an M4P project can embrace and deal with complexity in
monitoring and reporting on results. A TOC, which is linear, not externally vetted, and
which features assumptions that are not been adequately tested over time will make
it hard for the project to attribute change and thus demonstrate the web of market
structures and actors involved. This was argued in the case of GEMS 2 in the Lessons
Learning Review produced following the project’s closure. The report indicated that
the lack of a well-articulated strategic framework and gaps in its early analyses led to
disconnect between the log frame, TOC and intervention logics which in turn
contributed to GEMS 2’s under-performance.
The GEMS projects have been using internal benchmarking, in other words,
comparing against their own performance to analyse trends over the years. However,
in the economy aspect of VFM, where indicators have been standardized to a
considerable extent14, it has been possible to benchmark externally. The projects have
been compared externally in this way at least in the last two rounds of Annual Reviews,
including the May-June 2014 MTR. The standard economy indicators that have
emerged are:
Weighted average daily fee rate of consultants;
Overhead costs15 as percentage of total spending;
13 Improving VFM Measurement, Monitoring and Management: Economic Growth Portfolio Projects of DFID Nigeria, by ITAD,
May 2014.
14 There are some differences in how the GEMS projects apply economic indicators. For instance, whilst GEMS 4 only looks at the average rate of its short-term consultants, GEMS 1 and 3 take into account all short and long-term staff. Also, the exact definition of use of local versus international consultants varies across the projects. These differences were highlighted in the 2014 Mid-Term Review ARIES reports.
15 Defined as living costs, office space and other office running costs, flights, and other miscellaneous costs.
46
Breakdown between use of local and international consultants.
In terms of efficiency, it has been more challenging for the projects to identify indicators
that easily lend themselves to benchmarking. According to feedback from DFID-
funded programmes in Nigeria16, efficiency seems to be an area where identifying
appropriate indicators within a project is difficult, let alone comparing with other
projects. ‘Budget utilisation’ was used to compare efficiency across the GEMS projects
as part of the 2014 MTR. This is an indicator, which measures performance on
efficiency as it demonstrates and compares projects’ capacities in planning and
managing resources.
Other ways that the GEMS projects are reporting under efficiency, is to report
spending on project outputs (as in GEMS 1) or main interventions (as in GEMS 3 and
4), as well as spending apportioned across states that the project operates in (as in
GEMS 1 and 3). This type of information is helpful for showing costs associated with
a particular type of output, intervention or geographic location. It enhances our
understanding of costs involved in each case. However, spending breakdown by state
has little utility in terms of deciding where to invest. Factors that have to do with political
traction or equity (e.g. existence of high levels of poverty and/ or other vulnerability,
high incidence of violence, etc.) are more appropriate for shaping project decision-
making in this regard. A project may choose to work in Jigawa State for example even
though it may be more costly to operate there compared to another state based on
such factors.
GEMS projects should provide relevant narrative to explain, for instance, why cost per
woman beneficiary / reach may be low in a state (e.g. ‘many women are being
intentionally targeted and success is attributable to intervention X, which drove down
the cost per head’). A comparison between GEMS projects in terms of costs of working
in a particular state (e.g. Kano State) does not tell us much in terms of efficiencies of
those projects, simply because the nature of the work, operations and scope are too
dissimilar to make such comparison meaningful. On the other hand, spending on
outputs allows for comparisons externally as they enable such types of analysis as
‘calculation of cost per reach’.
However, ‘cost per reach’ or ‘cost per access’ and what these actually mean should
also be addressed in this context. Projects could better define what they mean by
reach or access. For instance, GEMS 3 considers the coverage of its sensitization
campaigns (e.g. on tax harmonization or business registration) through ‘cost per
beneficiary accessing products / services’ and reports this under efficiency. On the
other hand, ‘cost per beneficiary with an increase of 15% additional income’ is reported
under effectiveness (as an indication of cost-effectiveness) as the indicator is
concerned with a beneficiary reached with an intended outcome.
16 ‘Common Approach to VFM Reporting by DFID Nigeria State-Level Programmes’, IMEP, G. Bayaz & J. France, DFID Nigeria,
September 2013.
47
At cost effectiveness level, it is possible to make comparisons between GEMS
projects, as well as with other M4P projects as they use similar indicators such as ‘cost
per job created / obtained’; ‘cost per £ income generated’; ‘cost per £ of increased
income’. In addition, more detailed indicators such as ‘cost per £ of increased income
for the poor / for woman beneficiary’ aim to address equity considerations in
effectiveness. Each GEMS project also has an additional set of cost effectiveness
indicators, but to a large extend the above indicators are standard among all.
In terms of job creation, GEMS 1 has been limited by the cross-GEMS Results
Measurement Framework, in which the definition for a job has not squared with the
kinds of jobs that are common in the GEMS 1 sectors of meat and leather (i.e.
‘apprenticeships’). In the context of Nigeria’s meat and leather sectors,
apprenticeships are considered jobs. A recommendation coming out of the 2014 MTR
was that GEMS 1 count apprenticeships in their job numbers as this will create a more
complete picture of the value the project is generating given its particular context.
The recent ITAD study17 on DFID Nigeria’s economic growth portfolio programmes
argues that projects need to carry out more external benchmarking research. Only
GEMS 3 mentions in its reporting some programmes that can be looked at (e.g.
Katalyst in Bangladesh and Propcom Maikarfi in Nigeria).
Even if such research does not permit perfect comparative analysis, it will provide a
useful reference on how to tackle challenges common to M4P programmes such as
use of multiplier effects on income or jobs, or accounting for displacement and
deadweight loss.
All GEMS projects have VFM indicators that address equitable reach (pro-poor and
women) in their VFM frameworks and reporting. The costs reported so far are high
(higher than overall reach figures) as is often the case in development programmes.
The issue, however, of whether the poor benefit ultimately from the systemic changes
created in the markets that GEMS projects work in, remains to be more fully
addressed.
The main VFM challenges for the GEMS programmes and market system
development programmes more generally stem from timescale (i.e. it can take a long
to achieve impact, especially at scale) and attribution (i.e. it is hard to capture and
attribute systemic change when so many other factors are at play).
In terms of timescale, GEMS 1 and 3 are at the end of their fourth year of
implementation and have only recently started to record good progress on their cost
effectiveness indicators as a result of replication and scale-up (and in the case of
GEMS 1 crowding in and copying). All the GEMS projects have targets going two years
beyond the end of programming because the assumption is that it will need this
17 Improving VFM Measurement, Monitoring and Management: Economic Growth Portfolio Projects of DFID Nigeria, by ITAD,
May 2014.
48
additional timescale to achieve real value, and for the project to demonstrate continued
growth and improvements in markets beyond their lifetime – a key principle of the M4P
approach.
Some economists propose that a ‘ratings and weightings’ approach be used in order
to address the challenge of demonstrating results and VFM while value takes time to
generate. This approach has not been used so far in the analyses of VFM of the GEMS
projects as they face similar VFM requirements as other DFID programmes, which
requires showing results early on in the project cycle, including performance on
economy. The use of ratings and weightings could be further examined and
considered by DFID Nigeria.
All of the GEMS projects are following mixed methods to address issues of
attribution/contribution. GEMS 1 uses DCED compliance as validation and quality
assurance of these, and GEMS 3 use the Eminent Group18 mechanism. A systematic
review carried out for DFID19 of thirty M4P programme evaluations led to the following
recommendation as a good practice for evaluating attribution: ‘Evaluations should
….seek to ascertain the extent to which changes are systemic, carefully consider other
contributory factors, and additionally collect data over an appropriate period of time
following an intervention to capture long-term changes and measure sustainability’.
During the 2014 MTR, the Review team observed a good practice on attribution from
GEMS 1 in that they adjust beneficiary numbers and benefits occurred to account for
overlaps between interventions. Moreover, the DCED ‘mock audit’ found that GEMS
1 results accounted for displacement effects.
DFID Guidance on Economic Appraisals20 indicates that there are two main ways to
deal with attribution; one is through key stakeholder feedback and agreement; the
other is through pro rata cost contribution, if the benefits can be quantified. An example
for the first approach is how GEMS 3, in agreement with the World Bank, claims one
third of benefits from the Mortgage Financing Facility, which originated out of a study
funded by GEMS 3’s Flexible Facility. The second approach - pro rata cost contribution
- has not been used by GEMS projects so far.
A particular attribution challenge that was fed back to the Review team from the GEMS
projects was with regard to apportioning of results and costs among states. When an
intervention covers multiple states, it might be difficult to apportion results and
associated costs accurately between those states. Further improvements in M&E and
VFM frameworks, as undertaken by GEMS projects, may help overcome such
challenges in the future.
18 However, the MTR of GEMS 3 made reference to the fact that the Eminent Group are not M&E specialists and that an additional
external quality assurance/peer review process would reflect good practice for validation and quality assurance. 19 ‘Review of M4P Evaluations and Approaches’, DFID Working Paper, Ruffer T., Wach E., April 2013. 20 ‘How to Note on Economic Appraisals’, DFID, 2011.
49
4.4.2 Good Practice Examples of VFM Actions across GEMS Projects
In this section, good practice examples noted from across GEMS projects are
highlighted, with the particular projects indicated in brackets21. These examples can
generate learning for audiences beyond the GEMS projects – in particular other M4P
programmes in Nigeria and elsewhere.
Documenting negative lessons learned and reflecting these in project costs (GEMS
1). It is recommended for all projects that ‘failures’ be examined in a case study
format and reported.
Exploring the extent of an anticipated negative impact from a planned intervention:
e.g. assessing the impact on the livelihoods of traditional basket makers through
the introduction of plastic crates for transporting fresh tomatoes (GEMS 4).
Showing indirect costs and results measurement costs on each output (GEMS 1).
Training for staff on managing corrupt practices and anti-fraud (GEMS 1);
Use of service provider frameworks to ensure efficiencies in procurement across
sister projects (GEMS 3).
Use of DCED ‘mock’ audits and sharing of the findings of those audit reports with
project stakeholders (GEMS 1).
Sharing of administrative staff with ‘sister programmes’ in Nigeria (GEMS 3).
Systematic and periodic beneficiary feedback mechanisms such as a satisfaction
survey or viability index (GEMS 3).
4.4.3 Recommendations on Value for Money
There has been a steep learning curve for the GEMS projects with respect to
formulating their VFM frameworks and reporting. The content of both however is
getting more sophisticated as learning is accumulated and applied back into project
management. This evolutionary learning has been significant and very positive.
The project’s TOC is also highly relevant to understanding and interpreting the value
of a project. The more thorough the articulation of the theory of change, the more an
M4P project can deal with complexity in monitoring and reporting on results.
The GEMS projects have for the most part been using internal benchmarking to
analyse performance trends over the years. However, in the economy aspect of VFM,
where indicators have been standardized to a considerable extent, it has been
possible to benchmark externally. Recently, the ITAD study on DFID Nigeria’s
economic growth portfolio programmes pointed to the need for project to carry out
more external benchmarking research.
In terms of efficiency, it has been more challenging for the projects to identify indicators
that easily lend themselves to benchmarking. Efficiency seems to be an area where
21 More examples in this regard are provided in the 2014 MTR reports of each GEMS component.
50
identifying appropriate indicators within a project is difficult, let alone comparing with
other projects.
Comparative analysis on cost effectiveness is made simpler through the use of
standardized indicators such as ‘cost per job created / obtained’, ‘cost per £ income
generated’, and ‘cost per £ of increased income’.
The main VFM challenges for the GEMS programmes and market system
development programmes more generally stem from timescale for demonstrating
impact at scale and measuring attribution.
With respect to impact and VFM, the use of ratings and weightings could be further
examined and considered by DFID Nigeria as a means to address the challenge of
demonstrating results and VFM while value takes time to generate.
Reporting against standardized indicators has presented challenges in some cases.
For example costs per woman beneficiary reached will appear high in sectors where
fewer women are present; similarly, cost per job created is restricted to capturing FTE,
which is unrealistic given high levels of informal / piece work or apprenticeship roles
cross the GEMS sectors.
A quick scan across various M4P programmes indicates that the range of findings on
the cost per job created is remarkably wide; Even if such research does not permit
perfect comparative analysis, it can be a useful reference on how to tackle challenges
common to M4P programmes such as use of multiplier effects on income or jobs, or
accounting for displacement and deadweight loss.
There are a number of good practice examples of VFM in action across the GEMS
projects, which reflect their positive evolution in VFM analysis and reporting and which
could be usefully applied by a broader M4P audience.
4.5 Impact
4.5.1 Introduction
The findings and lessons presented in this section focus on exploring what lessons
have been learned on facilitating and maximizing sustained impact. There is clearly
overlap with findings in the other sections of this report since strategic framework, tools
and approaches, management and oversight, and robust M&E and VFM frameworks
are all fundamental components for maximizing M4P programming impact.
4.5.2 Lessons and Findings
This section is divided into two topics, Unintended Outcomes and Scale and Impact.
A. Unintended Outcomes
51
GEMS 1 and 3 provide good examples on the need for projects to continually scan for
and mitigate unintended negative outcomes and impacts, which can potentially be
damaging for intended beneficiaries.
In the case of GEMS 1, it identified several areas where it felt it could intervene within
the complex system of traditional abattoirs. Whilst some of these interventions have
been very positive; such as replacing manual inflation of skins on small ruminants with
air compressors; other planned interventions were revised as a result of the project’s
analysis and identification of potential adverse impacts. For example, the project
thought to install water taps near the abattoirs to reduce the difficult task of women
carrying water to the site (a task primarily undertaken by women). However, after
deeper analysis of the context, the project recognised that the installation of easily
accessible water would actually have excluded these women from the workforce. The
project redesigned the intervention to focus on other aspects of improving the health
and safety of women workers, including the supply of boots to protect their feet whilst
carrying loads, and also training in first aid. These adjustments to the intervention
strategy meant that the issue of displacement of labour for the very poor was avoided.
GEMS 3’s support to reform local business tax rates and payment systems (using
direct lodgement into local government accounts or point of payment systems) has
brought an end to large numbers of ‘illegal’ tax collectors operating in local markets
(these number in the hundreds in some cases). These illegal tax collectors are often
a source of harassment for micro and small scale entrepreneurs, and women
entrepreneurs especially, and so their removal from the system is a very positive
development. Local government response has been to re-train and re-employ some,
and to send small numbers on vocational education courses. In certain cases, GEMS
3 project management was not aware of the issue, and so had not been involved in
helping find appropriate solutions. However, without tactical monitoring of the situation,
there may be an unidentified risk to the newly introduced and more transparent tax
Lessons - Impact: Managing and Responding to Unintended Outcomes
With any systemic change, there is the potential for unintended positive or
negative impacts. It is the role of projects to understand and respond to these.
As interventions unfold, projects need to be able to sharpen their lens with
respect to how and what they monitor so unintended outcomes are identified and
appropriate responses made early on.
Project teams have to have the ability to remain flexible and responsive to
developments in the market place. This starts with project management style, but
even more critical is the right skills sets and expertise at field level.
Successful M4P intervention strategies are ultimately those that are driven by
the needs of market actors. Projects need to know when to get out of the way!
52
collection processes being put in place if for example these illegal collectors (and their
leadership) decide to push back. Roles may have changed in a short period of time,
but it is too early, and requires monitoring, to record eventual systemic change.
Both the GEMS 1 and GEMS 3 examples above illustrate how market system
development projects need to continually evolve and refine M&E tools as interventions
unfold and outcomes are increasingly beyond the direct control of the project. Projects
need to be able to sharpen their lens with respect to how and what they monitor so
unintended outcomes are identified and appropriate responses made early on and so
positive efforts do not unravel and undermine impact goals. Again, the importance of
a robust system for designing and using intervention logics and testing assumptions
as an integral part of intervention management comes to the fore.
Being able to provide strategic, catalytic and opportunistic support is a key strength of
the M4P approach as a means to maximize impact. Another common success factor
based on GEMS project experiences is that project teams have to have the ability to
remain flexible and responsive to developments in the market place. This starts with
project management style, but even more critical is the right skills sets and expertise
at field level.
For example, credit is seen as one of the limiting factors for market actors in the meat
and leather sectors in Nigeria. This is commonly encountered as a challenge in
development projects. For some years, GEMS 1 has been trying to find ways to
stimulate banks and other finance institutions to develop credit products suited to
agriculture. A solution came when GEMS 1 was able to demonstrate a clear business
case for why a bank should invest - with minimal project engineering.
Grassroots Bank was interested in expanding its agriculture portfolio, and attended a
GEMS 1 training; it became aware of the value of proper feeding of ruminants and the
increased returns small scale producers are achieving. At the same time, a group of
farmers and investors were exploring the possibility of starting a feedlot operation, and
intended doing this by pooling their financial resources and then seeking further credit.
They also attended GEMS 1 training and developed a strong business plan. GEMS 1
introduced the partners, and this led to a successful loan being established to finance
the start-up of the feedlot. This indicates that rather than trying to find ways to stimulate
banks and other finance institutions to develop credit products suited to agriculture,
the emergence of a clear business case meant the bank was willing to provide credit.
Once GEMS 1 introduced the two market players – they worked together to develop
a credit product to suit, and the business relationship moved forward. GEMS 1 as the
facilitator, using existing service providers with minimal intervention from the project is
a good example of M4P as an open and flexible approach to generating impact.
53
B. Scale up and Impact
GEMS 1 and GEMS 3 are both pursuing aggressive strategies for scaling up
successful interventions and activities into new states. In the case of GEMS 1, this
includes activities around feed finishing and finished leather being implemented in
three additional states in the north of the country. This has proven to be more difficult
due to the weaker capacity of the local actors when compared to earlier states. What
this demonstrates is that successful interventions rely heavily on local partners, and
such interventions therefore need to be designed with this in mind. If partners prove
to be slower in taking up interventions and activities than expected, plans need to be
amended quickly and respond to the local conditions.
GEMS 3 has faced similar challenges with respect to weaker capacities of its local
government partners in scale-up states. But it is also facing challenges in how its tax
reform model, which has proven successful in focal states (such as Lagos and Cross
River), is being applied in newer scale-up states, particular at grassroots level. The
project has developed and applies a toolkit approach to fast-track replication and
scale-up of its tax work stream, but there is a need to be aware of displacement issues
in any scale up activity. Such issues are highly contextual, however there is a need for
timely and regular review of intervention plans to ensure the model being replicated in
the new area is refined to meet what is in effect a new set of needs and circumstances.
Another emerging learning theme is the need for programmes like GEMS to have hold
of a well-articulated, much longer term vision not only for the sectors they are working
in but for the communities they strive to transform. Such a vision would take account
of developments that are highly likely to transpire regardless of project support. Such
an approach also involves looking at the ‘bigger picture’ of what is envisaged across
Lessons: Scale Up and Impact
Context is equally as important in designing (and refining) interventions as
the actual market system issue being addressed.
To maximize impact, scaling up needs to be based on ‘new’ intervention
plans that adequately reflect the context (recognising new needs and
circumstances) - not just repeating a successful model.
Over standardization of a model or toolkit designed to accelerate scale up
will lead to replication of weaknesses as well as strengths if appropriate and
timely reviews of intervention plans do not take place.
Projects benefit from more detailed longer term visions, not just for the
sectors they are working in but also for the communities they strive to
transform; such a vision could also pave the way for a flexible and innovative
model and tools and approaches that support more integrated delivery where
this makes sense.
54
sectors (cross sector impact as it were) as a result of M4P intervention support overall;
in the context of GEMS this could be better organised and capable women in well
function wholesale buying groups that make use of smart phone technologies, internet
access and growing distribution channels for profitable trading; these same women
would also be benefiting from access to appropriate financial services, improved
infrastructure, and a more conducive business environment. The question is whether
the current GEMS approach is able to deliver on such a vision. Given the current lack
of cross GEMS synergies (see section 4.4 on management and oversight and
experiences so far regarding synergies and collaboration across the GEMS projects)
this is likely to be by default rather than by design.
Adopting a bolder approach as described above links to earlier commentary on the
potential for using tools and approaches for economic modelling, sector modelling,
etc., and for further raising the bar with respect to M&E and VFM frameworks for
delivery and analysis; so development initiatives on the scale of GEMS deepen their
ability to help shape and project future market dynamics, whilst ensuring poorer
segments of society are participating and benefiting on an equitable basis.
The concepts touched on above cannot happen by default. They have to be envisaged
as part of programming vision, conceptualisation, and programme design, which will
then inform strategic framework, tools and approaches, skills sets needed over the
short and long term, etc. DFID Nigeria and the World Bank should consider a research
and development exercise that presents options/models for operationalizing a more
integrated GEMS approach, including what core elements such as strategic
framework, tools for design and analysis, and management and oversight would look
like at both programme and project levels.
There are several emerging learning themes around the topic of context and
maximizing impact. Firstly, projects should be careful not to be patronising of people
trying to make a living within a context; often context will suggest that for example
ecommerce is of no relevance, and yet in Nigeria widely affordable ‘Africa’ targeted
and designed smart phones are appearing, network coverage has expanded rapidly,
and increasing numbers of people are getting online to make use of social media and
internet based commercial platforms that are specifically targeting local domestic
markets (e.g. Dealdey, Jumia, Wakanow and ‘Kaymu’ - promoted as Nigeria’s own
‘eBay’, among others)22. Secondly, programmes like GEMS need to design and think
carefully with regard to context, but pay attention to likely trends outside the context,
so they (and those they are supporting) do not to get left behind by market
developments that the poor could be helped to benefit from sooner.
There are perhaps two ways we can look at context; firstly, what is traditional/cultural
and is blocking our efforts for change; secondly, what is emerging and new and can
22 Nigeria has seen a rapid surge in the development of ecommerce businesses. According to Euro-monitor international,
Nigerians are said to have spent N62 billion ($380 million) in online purchases as at 2011, with that figure predicted to rise as high as N150 billion ($920 million) by 2014; see ‘www.ventures-africa.com/2013/09/top-7-e-commerce-firms-nigeria’.
55
be leveraged in helping efforts for change? Context is a very general term that covers
all the positive and negative aspects surrounding our area of intervention; it is the skill
of the implementer in sorting these out that makes the difference with respect how and
what impact will be achieved.
4.5.3 Conclusions and Recommendations
To maximize impact, market system development projects need to continually evolve
and refine M&E tools as interventions unfold and outcomes are increasingly beyond
the direct control of the project. In order to do so, projects need to be able to sharpen
their lens with respect to how and what they monitor so unintended outcomes are
identified and appropriate responses made early on to ensure fragile or early stage
positive efforts do not unravel and undermine impact goals.
The opportunity to provide strategic, catalytic and opportunistic support is a key
strength of the M4P approach as a means to maximize impact. A common success
factor based on GEMS project experiences is that project teams have to have the
ability to remain flexible and responsive depending on developments in the market
place. This starts with project management style, but even more critical is the right
skills sets and expertise at field level in areas such as communications and
messaging, operationalizing, monitoring and feeding back on intervention logics and
assumptions, and monitoring for unintended outcomes – both positive and negative.
To maximize impact, scaling up needs to be based on ‘new’ intervention plans that
adequately reflect the context (recognising new needs and circumstances) - not just
repeating a successful model.
Moreover, over standardization of a model or toolkit designed to accelerate scale up
will lead to replication of weaknesses as well as strengths if appropriate and timely
reviews of intervention plans do not take place.
An emerging learning theme is the need for programmes like GEMS to have hold of a
well-articulated, much longer term vision not only for the sectors they are working in
but for the communities they strive to transform. Such an approach also involves
looking at the ‘bigger picture’ of what is envisaged across sectors as a result of M4P
intervention support overall. Given the lack of synergies, the current GEMS approach
is unlikely to be able to deliver on such a vision.
4.6 Sustainability
4.6.1 Introduction
This section explores what lessons have been learned about designing, implementing
and adjusting strategies to help ensure sustainability in market system development
programming. In essence, the previous sections present examples and lessons on
how the GEMS projects have gone about designing, monitoring, assessing, and re-
56
aligning strategies to ensure sustainability of results. Additional issues and lessons
that came out from the Lesson Learning Review process are explored below.
4.6.2 Lessons & Findings
Stimulating and enabling actors to improve the delivery of goods and services to the
poor is at the heart of the M4P approach. However, free market forces do not always
benefit the poor, so it is important for projects to test that changes in the market system
do not exclude the poor.
In GEMS 1, for example, the use of paravets to distribute feed in addition to providing
advisory services to farmers has been successful, but will this continue to benefit the
poor in the future? Paravets make money from selling feed, and in any free market
suppliers will normally sell where the highest profit can be achieved. In some rural
areas, this may mean selling to larger scale farmers near the paravets’ shops rather
than servicing small scale farmers a long way from their location. Sustainable solutions
that ensure more remote small scale farmers continue to receive adequate services
and products must be a priority for GEMS 1, in collaboration with local departments of
agriculture that continue to employ the paravets.
Another example of potential exclusion of the poor relates to the growing business
model of commercial feedlots. As these feedlots expand, particularly under operators
such as Master Meat that also have abattoirs and processing facilities, there is the
potential for small scale farmers/producers to be pushed out of the meat supply chain
as the lead firm develops its own cost effective supply chain solutions. For the supplier-
buyer business relationship to grow, it is important that the project find workable (and
viable) solutions that enable these dominant market actors (lead firms) to continually
do business with the small scale farmer. Otherwise, the poor will again be excluded.
In this example, it may be a solution to support a more formalised system of contract
farming to the larger actors. However, the key (and main driver of sustainability) will
be to allow the actors themselves to drive the solution.
Lessons: Sustainability
Free market forces do not always ensure the poor are benefiting from an
intervention. Projects need to clearly articulate the intervention logics and
desired results, and ensure activities drive sustainable outcomes
Adapting to market changes is important in an M4P context, but too much
change can be detrimental to sustainable change. Conversely, maintaining
effort in one intervention area can reduce the likelihood of market actors
developing the market themselves.
Selecting and working with the right partner is the most important element
of achieving sustainability
57
In the case of GEMS 4 it was suggested that too much adjustment to strategies in
response to market constraints was a weakness in achieving sustainable impact. In
this instance, asking why strategies are being adjusted and what is triggering
adjustment is what matters; changing strategies and intervention logics is very ‘M4P’
(in fact GEMS 1 did the very same and had to rework intervention logics several times
in some cases before it felt it had ‘got it right’); however, it is also for GEMS 4 to
ascertain that given its chosen approach, it has robust tools and mechanisms in place
to trigger when it needs to change direction and when it considers it is getting it right,
including ensuring solutions are being driven by the market actors themselves.
GEMS 3’s strategies for sustainability include embedding and institutionalizing tax,
land and investment reforms through high levels of participation of stakeholders in the
design process, and locking down improved rules and systems through legislative and
policy reforms at state and local government levels as part of its implementation
approach. However, GEMS 3 is equally aware of the need to inculcate systemic
change at the grassroots level for positive reforms to stay in place and continue once
GEMS 3 is no longer there. GEMS 3 strategy has therefore been to intentionally go
out and help create demand for reform through public sensitization and awareness
raising via BMOs, traditional leaders, and lead figures in business communities.
Demand creation also involves high involvement of local businesses in decision-
making, the use of theatre and public notices to reach broader community, and events
targeting women only entrepreneurs. It would need later evaluations of GEMS 3 work
(post project) to assess how robust these strategies have been with respect to shoring
up sustainability of reforms, but early signs show that creating noise in this way is
empowering the business community and bringing tangible benefits (such cost
savings, public works, or obtaining a certificate of occupancy) on the demand side,
and increasing pressure for responsible public service through greater transparency
and accountability on the supply side.
From the outset, the GEMS 1, 3 and 4 projects have each selected and built
partnerships with institutions and businesses in which there are senior individuals with
the right orientation and who appreciate the opportunities for tangible gains (both
actual and potential) that were on offer from the projects. All three projects have been
able to attract the right mix of partners; and these partners would likely all admit that
they are attracted to the GEMS programme because it is introducing services they
need, without being duplicative. GEMS 1, 3 and 4 are thus seen as not only being
innovative and flexible, but also as offering products and services that meet the current
needs and future outlook of its partners – and their constituents.
Moreover, the projects’ product offerings (such as training, technical expertise, market
facilitation, etc.) are not available in this form elsewhere in the market. Thus, the GEMS
projects are succeeding in serving as a “catalyst” by introducing interventions that are
addressing market needs, generating useful information and analysis on the extent
and nature of these needs, and involving the key stakeholders in planning and
58
implementation processes. A common success factor is the GEMS projects use of
‘social marketing’23 to generate win-win scenarios; however, on the receiving end it is
more often than not the incentive of financial benefits that is the deal maker.
4.6.3 Conclusions & Recommendations: Impact and Sustainability
At project level however, an M4P programme needs to be able to create its own niche
as an important pre-requisite for achieving sustainability. The development of a niche
(which is not the same as purpose or rationale) should begin at the point of programme
conception and evolve into the programme design and delivery strategies. A
sustainable M4P programme needs to introduce services that address market
constraints without being duplicative; its product offerings should be innovative and
flexible and have a ‘unique selling point’ in how they meet the current needs and future
outlook of partners.
Under an M4P approach adjusting strategies can be a sign of strength, but it can also
signal weaknesses if it continues for too long or there does not appear to be good
foundation for the change of plan; a project needs to be able to show (through its M&E
framework and how it reports for example) that it has robust tools and mechanisms in
place that trigger when it needs to change direction and when it considers it is getting
it right, including means for ensuring solutions are being driven by the market actors
themselves.
Programmes like GEMS need to think carefully with regard to context. Care should be
taken that approaches are not overly patronizing in how they assess and interpret
context; keeping a close eye on trends and developments outside the immediate
context will ensure projects do not get ‘left behind’ by market developments that with
the right support the poor could benefit from sooner. A good example across all GEMS
is the rapidly growing ecommerce space in Nigeria and ensuring the poor are included.
Always applying the lens of the poor – and the lens of women is critical to sustainable
outcomes. Projects should not assume that the benefits of successful interventions
reach the poor. As well as continually monitoring and evaluating market developments
and what systemic changes are emerging, projects need to test that changes in the
market system are not or will not exclude the poor.
It will be important for the GEMS projects to undergo an impact evaluation, which is
done at the appropriate time and is sufficiently formal and objective with regard to
deeper scrutiny of systemic change and the projects’ impact on the poor and women
especially, value for money, and to the extent possible the sustainability of positive
transformations and how this is evident in the targeted sectors, businesses and
23 Social marketing is the use of commercial marketing principles and techniques to improve the welfare of people and the
physical, social and economic environment in which they live. It is a carefully planned, long-term approach to changing human behavior; source: www.stdhivtraining.org/YSMT_socmarketing.html.
59
communities. Caution should be taken in assessing cross sector impacts however
since this does not appear to have been sufficiently programmed in from the
beginning.
5 Conclusions and Recommendations
This Lessons Learning Review report is based on information gathered from
engagement with the GEMS 1, 3 and 4 projects as part of the IMEP Mid-Term Review.
Workshops were also held with each project separately to gather information
specifically on lessons learned.
In addition to this, lessons taken from the review of GEMS 2 in 2013 also informed the
report. A review of project documentation and discussions with DFID and World Bank
representatives in Nigeria were also undertaken.
Recommendations
This Report provides a wide range of lessons that are summarised under each section.
From this, a smaller number of recommendations are provided, providing a brief
summary of the key aspects taken from this report that can be attributed to other
projects looking to take a Market System Development approach.
The list is intended as a guide to some of the aspects of the DFID supported GEMS
projects that can be developed for use in other programmes.
The first set of recommendations look at overall ‘programme’ management, and in
particular how a group of programmes can be managed. Following this are some
recommendations on tools and approaches.
1. Clear Vision (or Theory of Change) is Essential
Donors and service providers need to agree on a clearly articulated vision for the target
sector(s), or Theory of Change, at the outset of the programme. In the context of
GEMS suite, the difference between GEMS 1, with a clear vision of the target sectors,
and GEMS 2 that lacked this vision is clear. GEMS 3 and GEMS 4 had slower start up
phases whilst this vision was settled, but have since shown why ‘getting this right’ is
important. This vision also needs to be clear to project partners and stakeholders.
2. Front Loading of Market Analysis
Closely aligned to the above Recommendation, programme design must be informed
(and developed) based on sound market analysis. The successful interventions across
the GEMS projects supported this. Donors must allow projects to undertake adequate
market analysis early, so that interventions can be properly designed.
3. Use of Qualitative Indicators in log frames/reporting
Project management log frames should not only include quantitative indicators, but
also qualitative indicators that adequately reflect qualitative targets for systemic
60
changes and impacts; another option could be to introduce a results measurement
framework that drills down on qualitative (and quantitative) project targets and goals,
and which complements the conventional log frame template currently used. This was
used effectively by, in particular, GEMS 1.
4. Annual Reviews
DFID should also consider how the Annual Review process (including content of the
ARIES template) could be more useful for more detailed analysis of the evolution and
status of the log frame indicators and targets to ensure these are well aligned with the
TOC, results chains and intervention strategies.
5. Cross Programme Collaboration
The GEMS programme was conceived as a suite of interacting projects. In this
context, the GEMS suite had both a cross-GEMS log frame, but also expectations of
closer collaboration.
However, the structure and consistency applied across projects in terms of reviewing
and revising log frame targets and also intervention level outputs and targets was
inconsistent.
The resulting lack of coordination in the log frames meant collaboration was less
effective. to ensure log frames remain relevant and useful as M4P programme
management tools, more consideration needs to be given to timing of adjustments,
closer scrutiny of rationale for adjustments, and a documented record of changes.
6. Use of DCED Standards
Irrespective of the approach being used (M4P, BEIF, Value Chains) the use of the
DCED standards for results measurement is considered to be best practice. GEMS 1
felt that using these standards, as well as the Mock Audit, allowed them to better
manage project outcomes, leading to improved impact.
7. Results Measurement (the Handbook)
The GEMS projects were intended to use a standardized approach to results
measurement, ‘the Handbook’.
If this approach across a suite of programmes is to be used, more attention on ways
to reflect on the usefulness of the handbook and to review and upgrade the
approaches. Despite some limitation on the use of the Handbook, there is an
opportunity for the GEMS projects to share more broadly on the benefits of applying
the approach, and their experiences on practical adaptation and application to
maximize effectiveness in M4P intervention design, management and delivery.
8. Responding to Unintended Outcomes
Whether positive or negative, unintended outcomes or results are likely to occur over
the course of a programme. Allowing the flexibility in the initial project design to
respond to these is important. Both GEMS 1 and GEMS 3 had examples of where this
occurred, and they were able to capitalise on results.
61
9. Use of PPEM & PPD
The public private engagement mechanism (PPEM) and public private dialogue (PPD)
approaches applied by GEMS 3 are both highly successful components of the project’s
strategic framework and would be useful in the context of other market system
development programmes. DFID is encouraged to request GEMS 3 to document in
more detail the model/approach used to enter and engage with project stakeholders.
10. Embedding technical expertise with partners
Both GEMS 1 and 3 effectively used the model/approach of embedding technical
expertise from the project with partners. There are aspects of this approach that has
helped the projects succeed, and which would be useful learning for a broader market
system development programme audience. This should be better defined and
documented by the projects, so it can be disseminated to others.
11. Innovative Tools and Approaches
The above examples show that innovative ways of project implementation should be
encouraged. Donors, including DFID, should support research and development for
introducing more innovative tools and approaches for M4P design and delivery; for
example, increased use of evolving ‘business’ cases, business modelling, broader
sector / economic modelling, and commercial benchmarking.
Implicit in this recommendation is that successful projects will use a range of delivery
mechanisms, and not be restricted to working within one approach only. The context
in which projects’ operate normally requires an adaptive approach.
12. Value for Money
With respect to VFM and impact, the use of ratings and weightings could be further
examined and considered by DFID Nigeria, and other donors, as a means to address
the challenge of demonstrating results and VFM while value takes time to generate.
Dissemination
The primary recipients of the findings of the Lesson Learning Review are DFID Nigeria
and its M4P service providers as well as DFID staff engaged in M4P programmes in
other countries. These lessons can also be disseminated to the broader community of
M4P programme funders and practitioners that can benefit from the Review lessons
on implementation of M4P projects.
In addition to the planned presentations and reporting deliverables, the final Lesson
Learning Review report (or an abridged ‘Note from the Field’) will be distributed
internally in DFID Nigeria and across DFID’s community of M4P projects in Nigeria as
well as through the broader DFID network of M4P programmes. As per DFID policy,
the report will also be made available to the public via the DFID website.
IMEP together with DFID Nigeria and the Review Reference Group will also explore
innovative and cost-effective ways of sharing the lessons learned from the Lesson
62
Learning Review. This could include posting of one or more field notes on existing or
new websites, or web-based ‘hubs’ or resource centres, which are recognised forums
for M4P learning. Other options include: condensing learning into a set of principles
for publication; supporting or contributing to a series of online webinars that present
and discuss findings; and facilitating the availability of material for bigger, more
comprehensive evaluation exercises being conducted with support from DFID.
63
Annex 1: Summary Profiles of the GEMS Projects
Growth and Employment in States (GEMS) is one of a suite of programmes funded by
the United Kingdom (UK) Department for International Development (DFID) and the
World Bank (WB) whose aim is to help reduce poverty in Nigeria through increased
economic growth, incomes, and jobs in selected states. Four GEMS components have
been implemented to date and profiles are summarised as follows.
GEMS 1 Meat and Leather
GEMS 1 started in-country operations on 10th March 2010. Its focus is on improving
the livelihoods of poor men and women in the meat and leather sector by applying a
“Making Markets Work for the Poor” (M4P) approach. The Market Sector interventions
are summarised as follows:
Market Sector
Support
Outputs / Deliverables through GEMS1 facilitation
Livestock Feeding 1. New and/or improved inputs, products and services benefiting
poor people within the livestock feeding markets are introduced;
Meat Processing 2. New and/or improved inputs, services and products that benefit
poor people within the meat processing markets are introduced;
Finished Leather
Goods
3. New and/or improved inputs, products and services that benefit
poor people within the finished leather goods markets are
introduced;
Finished Leather 4. New and/or improved inputs, products and services that benefit
poor people within the finished leather market are introduced;
Skins Supply 5. New and/or improved inputs, products and services that benefit
poor people within the skin supply market are introduced;
Organisation and
Advocacy
6. Meat and leather industry competitiveness improved through
enhanced organisation and advocacy skills (capacity) in the
sectors;
Financial Products 7. Meat and leather industry competitiveness improved through
enhanced use and management of new/and or improved financial
products available in the sector
In late 2013, GEMS 1 and DFID agreed on Contract Amendment 2 in which GEMS 1
was allocated a revised budget of £8,789,963 over the project period. This budget was
agreed based on increased log frame targets. Contract Amendment 2 allowed for
additional resources to be employed by the project, with additional technical staff in
64
most intervention areas, an additional person for results measurement, and more time
allocated for the Technical Director to support scale up initiatives. GEMS 1 now
operates across the following states: Abuja, Aba, Lagos, Kaduna, Kano and new scale
up states in the north, Jigawa, Zamfara and Katsina. GEMS 1 is due to finish on 1st
September 2015.
GEMS 2 Construction and Real Estate
GEMS 2 overall objective was to strengthen the performance of market systems in the
construction and real estate sector in order for these to perform more effectively and
offer increased opportunities (jobs and income) for the poor, and for women in
particular in the sector. GEMS 2 began its activities in April 2010 and ceased
operations at the end of December 2013. The GEMS 2 project was anticipated to run
from early 2010 until 2015. However, DFID Nigeria took the decision to close the
project early based on their assessment that it was unlikely to achieve its objectives
within the remaining project lifetime. As commissioned by DFID, a Lesson Learning
Review took place from 2nd to 6th December 2013. The findings and recommendations
were presented to DFID in a final report in May 2014.
GEMS 3 Business Environment
The mission of GEMS 3 is to “work with private and public sector stakeholders at
national, state and local government levels to build and deliver a systematic framework
that will help make it easier to do business in Nigeria, leading to lasting opportunities
for the poor, especially women”. The GEMS 3 project currently works in nine Nigerian
states. The project’s key turnaround point came in September 2012 when it designed
and adopted a systematic approach to improving the business enabling environment
through what GEMS 3 refers to as its Business Environment Improvement Framework
(BEIF). GEMS 3 began by piloting its BEIF approach in the Federal Capital Territory
(FCT) and four focal states (Lagos, Cross River, Kano, and Kaduna) before diffusing
the approach to four scale-up states (Jigawa, Kogi, Zamfara and Katsina) from July
2013. At the time of the MTR, GEMS 3 was still in consultation with DFID Nigeria as
to whether original plans to expand to an additional four States is a viable option in the
future.
Key intervention areas are outlined as below:
Output 1 Value adding business services and "products” (i.e., policies, rules and regulations) addressing "Tax System" constraints for target enterprises and firms are developed and delivered
Output 2 Value adding business services and "products” (i.e., policies, strategies) addressing "Land" constraints for target enterprises and firms are identified and strengthened
65
Output 3 Value adding business services and "products” (i.e., policies, strategies) addressing "Investment" constraints for target firms and enterprises are identified and strengthened
Output 4 Evidence based and joined-up policy dialogue initiated and promoted
Output 5 Non focal states and other projects demonstrating interest in enhancing BE
GEMS 3 is currently undergoing a second Contract Amendment (CA2). Whilst keeping
the overall contract value constant, DFID Nigeria has endorsed an additional spend of
GBP 3.0 million, which was to be achieved before end March 2014. This increased
rate of expenditure is being proposed in parallel with a third contract amendment,
which, if agreed by DFID, will be used by GEMS 3 to extend the Flexible Facility to
support increased impact, consolidation, and scale-up through until 2017.
GEMS 4 Wholesale and Retail Market System
Component 4 of the GEMS suite of programmes (GEMS-4) started operations on
August 1, 2012 with a nine and a half months inception period. A revised log frame for
the programme was approved by DFID in January 2014, which forms the basis of this
Mid-Term Review.
The focus of GEMS-4 is on improving the livelihoods of the poor and women in the
wholesale and retail sector by applying a “Making Markets Work for the Poor” (M4P)
approach. Although GEMS-4 interventions are intended to affect the Wholesale and
Retail Market System (WRMS), its interventions have so far, by design, not been
directed at the primary producer level (e.g. farmer level) within the sector.24GEMS-4’s
four initial interventions are:
Good handling practices in perishable produce, which mainly involves efforts to upgrade collection centres into packing houses and introduction of the use of plastic crates in the transportation of tomatoes from the main production centres to the final consumption areas.
Financial solutions for payments in supply chains, which involves the establishment of Mobile Money (MM) agent networks and Business to Business (B2B) payments in the horticultural sector.
Micro-distribution involving the formation and facilitation of Wholesale Buying Groups (WBGs) to distribute solar lamps.
Introduction of a Challenge Fund aimed at providing grants to firms to develop and pilot innovative solutions with a pro-poor impact, especially impact on women.
24 DFID has, however, recently approved the conduct of value-chain analysis of the horticulture and rice sectors aimed at
identifying backward and forward linkages with the farm sector and assessing suitable entry points for additional engagement with farmers in the two sectors.
66
In addition to the above four initial interventions, ten other interventions were identified
as candidates for further studies and potential design, which could be considered as
pipeline interventions. Value chain analysis of the horticulture sector has been
conducted and the programme has also conducted a value chain analysis of the rice
sector to study its demand structure and the supply links between farmers, millers,
retailers and consumers.
GEMS-4 presently operates in three of the four states that were originally targeted:
Kano, Kaduna, and Lagos. Activities are yet to start in Cross River State but there are
plans to expand beyond the originally targeted four states into the trading centres of
Aba in Abia State and Onitsha in Anambra State.
Following the start-up of the programme, DFID proposed a £3 million spending cap for
the 2014-2015 financial year as against a £4.36 million budget submitted by the
programme. The spending cap is expected to result in the scaling down of some
project activities as well as in a review of the programme’s mix of interventions. The
review team has made comments on the prioritized work plan proposed by GEMS-4
to adjust to the proposed budget ceiling. The comments are added to this review as
Annex 1.
67
Annex 3: Stakeholders, Governance and Process Issues
Stakeholders
The primary stakeholders for the GEMS 1, 3 and 4 MTRs include DFID, the Federal
and State Level Governments of Nigeria, the WB and the GEMS project teams.
Additional stakeholders include project partners such as federal and state level
ministries, local government departments, public institutions, private companies,
representative bodies (such as Business Management Organisations (BMOs)), and
business owners and their employees. Accountability extends to the UK taxpayers and
Nigerian people that are the providers and/or recipients of the UK Aid assistance.
The Lesson Learning Evaluation is intended to be useful to primary stakeholders such
as DFID Nigeria and its implementing partners, as well as a wide spectrum of
stakeholders such as DFID globally, and funders and implementing agencies of M4P
and market system development programmes more broadly.
Governance
The Evaluation design has been developed in close consultation with DFID Nigeria,
including the Results, Evaluation and Statistics Advisor, Esther Forgan, and the
respective Lead Advisors for the GEMS components at the time – Andrew Gartside
(GEMS 1 and 4) and Robert Hale (GEMS 3). Based in Abuja, these DFID personnel
acted as principal resource persons to the Evaluation Team to support formulation of
a valid TOR, provide guidance on and oversight of the Evaluation design, and ensure
clear direction for the Evaluation.
The Evaluation was organised and managed by the Independent Monitoring and
Evaluation Project (IMEP), which is a five-year project focused on the provision of
monitoring and evaluation services to DFID. IMEP covers the suite of State Level
Programmes in Nigeria, including GEMS, and is implemented by a consortium of
companies led by ECORYS. The IMEP Project Manager is Gregor MacKinnon.
An Evaluation Reference Group (ERG) has been established. The role of the ERG
has been to help to steer and advise the independent Evaluation at key strategic
points. A particular role of the ERG is to advise on maximising the usefulness and use
of this Evaluation. In addition, the ERG will continue to review and comment on key
outputs from the Evaluation, bringing in sector and practitioner views as appropriate,
and advising on the communications and dissemination strategy. The role of the ERG
has been advisory as opposed to executive. The ERG has been chaired by Esther
Forgan, the DFID Nigeria Results, Evaluation and Statistics Advisor, and its other
members include: Adrian Stone (DFID); Simon Calvert (DFID); and Andrew Gartside
(DFID Nigeria).
DFID External Quality Assurance (EQA) has also been involved in reviewing and
conducting quality assurance on the Evaluation design document on entry, and this
68
Evaluation report on exit, in line with normal DFID quality assurance processes. A
member from DFID’s external QA panel (SEQAS) was appointed to conduct the EQA.
As part of its mandate, IMEP regularly engages with and builds capacity within key
GON line Ministries and departments by involving nominated representatives as
appropriate in evaluation and review missions. In the case of this joint MTR and
Lesson Learning Evaluation, representatives from the Ministry of Trade (MOT) and the
National Planning Commission (NPC) participated as observers only - primarily to
ensure a fully independent and objective review process. Similarly, though
Government officials were frequently involved in stakeholder consultations during the
MTR, they played a limited role in the actual analysis process. However, the GEMS
project teams closely involve relevant Ministries, Departments and Agencies (MDAs)
in project planning and implementation, which is the MDAs main entry point for
supporting programming dialogue and decision-making.
Process Issues
In keeping with its role, IMEP acted as an impartial evaluator with no major conflict of
interest in the evaluation of the GEMS programme. In addition to the governance
framework and oversight described above the Evaluation team was able to capture
the views of a wide variety of stakeholders during implementation.
The Evaluation followed the Organisation for Economic Development (OECD)
Development Assistance Committee (DAC)25 evaluation principles and quality
standards with respect to aid effectiveness and evaluation of developmental
programmes, as well as DFID evaluation policy. The GEMS Results Measurement
Handbook (based on the Donor Committee on Enterprise Development (DCED)
Standard for Results Measurement) was a key point of reference for the MTRs.
The Evaluation also sought to comply with DFID’s stated Ethical Principles for
Research and Evaluation in the following ways:
adopting and adhering to a ‘do no harm’ policy with respect to entry, presence and engagement with local communities, including an awareness and sensitivity that data collection for the MTR was being undertaken in volatile and conflict affected communities;
respondents’ confidentiality was respected at all times and interview procedures respected local norms and customs, including, for example, appropriate meetings and engagement with local government authorities, community leaders and traditional leadership to help arrange and ensure appropriate participation in group meetings, and complying with and ensuring the informed consent of respondents approached during the fieldwork;
gender-specific focus groups were undertaken based on informed guidance from the GEMS field staff and their local contacts on appropriate measures for effective participation of women; and
25 This standard can be viewed in more detail at http://www.oecd.org/dac/
69
safety of the evaluation team was managed as part of IMEP’s existing security policy.
The MTRs were also governed by the general policy of IMEP for such reviews, which
stipulate that in the event that IMEP feels that a Lead Advisor for the programme is
having an undue influence on IMEP’s findings or recommendations it would
communicate this to the Results Advisor and Deputy Head of DFID Nigeria to resolve
any issues. Similarly, if any Lead Advisor for GEMS is concerned about the quality or
objectivity of IMEP reviews then s/he can have recourse to the Results Advisor and
Deputy Head of DFID Nigeria to resolve such issues. However, no such circumstances
were encountered by the key parties involved.
Consideration was given during the preparation phase to recording stakeholder
meetings and interviews. In the end this did not take place as the structure and
processes were deemed adequate for information gathering and analysis. It was also
felt that necessary explanations and seeking permissions would likely impede more
natural and accepted meeting practices across the different levels of stakeholders.
Dissemination
The primary recipients of the findings of the MTRs are DFID Nigeria, the World Bank
Nigeria office, the GON (at federal and state levels) and the GEMS project teams.
The primary recipients of the findings of the Lesson Learning Evaluation are DFID
Nigeria and its M4P service providers as well as DFID staff engaged in M4P
programmes in other countries and the broader community of M4P programme
funders and practitioners that can benefit from the Evaluation lessons on
implementation of M4P projects.
In addition to the planned presentations and reporting deliverables, the final Lesson
Learning Evaluation report (or an abridged ‘Note from the Field’) will be distributed
internally in DFID Nigeria and across DFID’s community of M4P projects in Nigeria as
well as through the broader DFID network of M4P programmes. As per DFID policy,
the report will also be made available to the public via the DFID website.
IMEP together with DFID Nigeria and the ERG will also explore innovative and cost-
effective ways of sharing the lessons learned from the Lesson Learning Evaluation.
This could include posting of one or more field notes on existing or new websites, or
web-based ‘hubs’ or resource centres, which are recognised forums for M4P learning.
Other options include: condensing learning into a set of principles for publication;
supporting or contributing to a series of online webinars that present and discuss
findings; and facilitating the availability of material for bigger, more comprehensive
evaluation exercises being conducted with support from DFID.
70
Annex 4: MTR and Lessons Learning Evaluation Framework
Attached as separate document
71
Annex 5: Project Documentation Used for Review and Analysis
GEMS1
1. GEMS 1 – Meat & Leather: Inception Report, October 2010
2. GEMS 1 – MEAT & LEATHER: Mid Term Report 2014 (July 2013 to March
2014 – 9 months), May 2014
3. GEMS 1 Revised Log frame, Contract Amendment 2, May 2014
4. Independent Monitoring and Evaluation Project for the State Level Programmes
(IMEP) Annual Review Growth and Employment in the States
2013, July 2013.
5. GEMS 1 – MEAT & LEATHER - Period Report 2013 (DRAFT FINAL- 23rd
May) July 2012 to May 2013 – 11 months, May 2013
6. Annual and quarterly reports for 2012 and 2013
7. Intervention reports and sector reports provided by GEMS 1
8. GEMS 1 ARIES Report, June 2014
GEMS 2
GEMS 2 Lesson Learning Review Report, IMEP, May 2014
GEMS 3
1. GEMS3 Annual Report 2013:
GEMS3 Annual Report 2013 - Executive Summary
GEMS3 Annual Report 2013 - Part A – Narrative GEMS3 Annual Report 2013 - Part B - Monitoring and Evaluation Results Progress Report
GEMS3 Annual Report 2013 - Part C - Value for Money Report
GEMS3 Annual Report 2013 - Part D - Scaling Up Plan (Non-focal States)
GEMS3 Annual Report 2013 - Part E - Finance and Management Accounts
2. GEMS3 Annual Report Appendices
GEMS3 Annual Report 2013 - Part A Appendix A - Organisation Structure
GEMS3 Annual Report 2013 - Part A Appendix B - Consolidated Responses to Annual Report 2012 Recommendations
GEMS3 Annual Report 2013 - Part A Appendix C - Consolidated Responses to World Bank Support Mission
GEMS3 Annual Report 2013 - Part A Appendix D - Mission, Vision, Strategy
GEMS3 Annual Report 2013 - Part A Appendix E - Business Processes Manual
GEMS3 Annual Report 2013 - Part A Appendix F - Kano Business Environment Improvement Strategy
72
GEMS3 Annual Report 2013 - Part A Appendix G - Kaduna Business Environment Improvement Strategy
GEMS3 Annual Report 2013 - Part A Appendix H - Cross River Business Environment Improvement Strategy
GEMS3 Annual Report 2013 - Part A Appendix I - Lagos Business Environment Improvement Strategy.pdf
GEMS3 Annual Report 2013 - Part A Appendix J - Federal Business Environment Improvement Strategy
GEMS3 Annual Report 2013 - Part A Appendix K - Project Plan
GEMS3 Annual Report 2013 - Part A Appendix L - Risk Register
GEMS3 Annual Report 2013 - Part A Appendix M - Documents to Share with IMEP
GEMS3 Annual Report 2013 - Part B Appendix A - Logframe Achievements
GEMS3 Annual Report 2013 - Part B Appendix B - Data Collection Plan
3. GEMS3 Quarterly Report Q4 2013
GEMS3 Quarterly Report Q4 2013 Annexes
GEMS3 Quarterly Report Q4 2013
4, GEMS3 Logframe revised May 2013 5.GEMS M&E Handbook 6.Guidace Note on measurement System 7.Guidance Note on indicators 8.Baseline study Tax Kaduna
9.Baseline study Tax Kano
10.Baseline study Tax Cross Rivers
11. Baseline study Land Kaduna 12.GEMS3 Inception Report
13. GEMS3 Inception report – Annexes
14. GEMS3 Annual Report 2014:
GEMS3 Annual Report 2014 PART 1 Executive Summary
GEMS3 Annual Report 2014 PART 2 Milestones Achieved Summary
GEMS3 Annual Report 2014 PART 3 Main Report
GEMS3 Annual Report 2014 PART 4 M&E Report
GEMS3 Annual Report 2014 PART 5 Value for Money Report
GEMS3 Annual Report 2014 PART 6 Finance and Management Accounts
GEMS3 Annual Report 2014 PART 7 Risks Table
73
GEMS3 Annual Report 2014 PART 8 Organograms
15. GEMS3 Annual Report 2014 M&E Annexes:
M&E Annex 1 GEMS3 RC Business Registration
M&E Annex 1 GEMS3 RC Land Registration
M&E Annex 1 GEMS3 RC Tax Harmonisation
M&E Annex 2 Outputs 5 results
M&E Annex 3 Project Impact FY2014
M&E Annex 4 Logframe
16. Monitoring Reports and Strategy Documents
MoUs : Kogi, JSG, Zamfara WEE: WEE support plan, Jigawa WEE FGD Report, Kogi WEE FDG Report, Wee Guide
BEIS: Jigawa, Katsina, Kogi, Zamfara
PPEM: Katsina, Kogi, Jigawa, Zamfara
Needs Assessment: Kogi
Selection of Intervention Plans
- Kogi Tax
- Jigawa SLTR
- Jigawa Tax
17. Studies and Reports
- Kogi SCPZ
- WEE success stories
18. Communications
- Guardian: SLTR article
- Links to Media Bites and Coverage
- Vanguard daily: Tax reform in Kogi
19. GEMS3 Business Process Manual.v0514
20. GEMS3 Tax Activity Guides
Complaints Process Activity Guide
Harmonised Tax Laws Activity Guide
Improved Payment Systems Activity Guide
Point of Sale (PoS) Payments Activity Guide
Sensitisation Activity Guide
Tax for Service Activity Guide
Tax Toolkit Activity Guides
Training Activity Guide
21. Indices
74
Perception Index Questionnaire
Viability Index questionnaire
GEMS 4
1 M&E Framework
2 Results Measurement for the Market Modernization Fund
3 GEMS4 Logframe
4 GEMS4 Quarterly Progress Report 1
5 Value for Money Strategy
6 Challenge Fund Documents
7 GEMS4 Quarterly Progress Report 2
8 GEMS4 Provisional Annual Report
Design Documents
1 Programme Memorandum
2 Terms of Reference
3 Technical Proposal
4 Inception Report
5 Inception Review Report
6 Inception Review Report Response
7 Status of Recommendations from Inception Review Report
Market Analysis Reports/Studies
8 Market Overview Report
9 Market System Analysis
10 Wholesale and Retail Sector Market Overview
11 WRMS Perception of Policy and Advocacy Needs Assessment
12 Consumer Preferences in Wholesale Retail Sector
13 Investigating Financial Services Opportunities in the Wholesale & Retail Sector
14 Research on Nigeria's Export to the UK
15 Investigating the Use of Mobile Money in the Wholesale and Retail Sectors of
Nigeria
16 Political Economy Phase 1 - Desk Review of Rules of the Game
17 Political Economy Study on Market Places and Market Systems, Governance,
Formal and Informal Rules and Civil Society
75
18 Initial Stakeholder Mapping of WRMS
19 Political Economy of Retail and Wholesale Sector II - Trends and Power
Relations in the Market Place
20 Final Report on Physical Market Infrastructure in Key Markets in Aba, Kano and
Lagos. Nigeria
21 Political Risk Analysis
22 Wholesale and Retail Supporting Actors Study
23 A Study on Vulnerable Groups, Gender & Poverty
Reports/Studies
1 Tomato Production and Marketing - Value Chain Analysis
2 Action Research Report - Tomato Packing and Transportation
3 Market Profiling
4 GEMS4 Post Harvest Training Evaluation Report
5 Assessment of Potential Displacement Effect on Basket Market by Introduction
of Plastic Crates to the Tomato Supply Chain
6 GEMS4 Wholesale and Retail Market Structure Mobile Money Pilot Assessment
7a Baseline Study on Horticulture - Qualitative
7b Baseline Study on Horticulture - Quantitative
8a Baseline Study on Mobile Money - Qualitative
8b Baseline Study on Mobile Money - Quantitative
9 Baseline Study on Micro-distribution
10 Market Enumeration of Wholesalers and Retailers in Key Hub Markets
List of Additional Documents
• Evidence of Reported Results
• Evidence for Output 1
• GEMS4 Annual Review 2014 Output 1 Summary
• Documented Evidence for Indicator 1.1
• Documented Evidence for Indicator 1.2
• Evidence for Output 2
• GEMS4 Annual Review Output 2 Summary
• Documented Evidence for Indicator 2.1
• Documented Evidence for Indicator 2.2
76
• Documented Evidence for Indicator 2.3
• Supplementary Evidence Catalogue
• Intervention Plans
• Intervention WEE Stories
• Market Modernization Challenge Fund Documents
• Concept Note Forms
• Final Application Forms
• Term Sheets
• Challenge Fund Concept Note
• Operational Manual and Policies
• Result Measurement Framework
• Monitoring and Evaluation Plans
• Response to Recommendations from Inception Report Review
• Annex Documents
• Impact Evaluation Workplan
• M&E Framework
• M&E Manual
• Mid-term Review Presentation
• Results Chain
• Results Dashboard
• VFM Strategy
• Logframe
77
Annex 6: Lesson Learning Workshops – Outline
• Session 1. Sustainability and Impact
What lessons have been learned about designing, implementing and adjusting
strategies to help ensure sustainability (focus on pro poor, WEE);
What can we learn about gathering evidence at market level and using this for
adjusting strategies? e.g. for facilitating and maximizing sustained impact?
Strengths and weaknesses of the M4P approach in this regard?
Observations on alternative approaches for achieving impact?
Should we promote indicators that capture systemic changes/non-economic
changes?
Should scaling up entail “Deeper” or “Broader”?
Spin offs should help create more spin offs;
• Session 2. Management & Oversight
What has worked and what has been learned with respect to a ‘facilitative
approach’ to partner/intervention management?
What has worked and what has been learned with respect to partner selection?
What have you learned with regard to intervention design and management
and sequencing with M&E system?
VFM: What lessons have been learned about implementing M4P programmes
to maximize the potential to increase impact and VFM? Particularly, recognising
that value takes time to deliver at scale, and that attribution may be harder to
make in M4P programmes;
• Session 3. Is M4P the right approach …
In which areas and how have the projects been innovative in their market
development approaches/methodologies; Are there principles and practices for
future application?
What, if any, are the trade-offs programmes have made in endeavouring to
implement ‘the ideal M4P approach’ in a practical context?
What are the lessons learned in designing the strategic framework for an M4P
project for achieving pro-poor systemic change?
Has context been used to design interventions or has it been used to explain
difficulty in implementing interventions already designed?
78
• Session 4. Learning from Failures/Success
Insights to be drawn from failures and successes that can inform future
initiatives;
Closer examination and discussion on successes, failures and what underlies
both, project responsiveness to changing circumstances;
What lessons can we draw (DFID/service providers) in terms of translating the
theory of change into an effective delivery strategy? What could be done
differently?
