Measuring Results in Challenge Funds

Practical Guidelines

for Implementing the DCED Standard

1

By Adam Kessler

October 2013

Preface

The Donor Committee for Enterprise Development (DCED) is the long-standing forum for donors,

foundations and UN agencies working in private sector development (PSD), who share their practical

experience and identify innovations and formulate guidance on effective practice. For more

information on the DCED, see www.Enterprise-Development.org

The Committee has been actively working on results measurement for some years, under the

guidance of the Results Measurement Working Group (RM WG), leading to the articulation of the

DCED’s Standard for Results Measurement. The Standard initially found most enthusiastic

application in programmes that make markets work for the poor (M4P) and the RM WG has been

keen to see it also applied in other contexts.

That process is gathering momentum. The Working Group for PSD in Conflict-Affected Environments

has published Practical Guidelines for Implementing the DCED Standard in CAEs; the Business

Environment Working Group has similarly published Practical Guidance on measuring donor-

supported business environment results, using the approach codified in the Standard. Other

applications are in the works.

Several Challenge Funds have started to use the DCED Standard in their work, and this synthesis of

their experience is therefore a logical next step for DCED. As many DCED members are becoming

more active in partnering with the private sector for PSD, a DCED Task Force is being formed at the

time of writing.

Acknowledgements

The author is indebted to interviewees and reviewers of the early draft of this report. Thanks are

due to Mark Thomas, Amanda Jupp, Wafa Hafiz, Marieke de Ruijter de Wildt, Nico Mensink, Ellen

Carey, Tim Ruffer, Edward Hedley, Alexandra Mandelbaum, Aly Breedlove, Eliguard Dawson,

Buddhika Samarasinghe, James Carnegie, Millycent Aoko, Kristian Holmberg, Corin Mitchell,

Alexandra Miehlbradt, Ben Fowler, Mihaela Balan, Erin Markel, David Smith and Caroline Ashley for

their comprehensive and incisive feedback. In particular, this guidance benefitted immeasurably from

the support and advice of Melina Heinrich, Nabanita Sen, and Jim Tanburn from the DCED

Secretariat.

Much of the advice in these guidelines is drawn from innovative challenge funds and organisations

working to strengthen their results measurement. In particular, I would like to mention the African

Enterprise Challenge Fund (AECF), Enterprise Challenge Fund (ECF), Food Retail Industry Challenge

Fund (FRICH), Business Innovation Facility (BIF), and Vietnam Challenge Fund. I also received valuable

input from representatives of ITAD, Triple Line, IRIS, and MarketShare Associates.

Front page photo credits: Mass Communications Specialist 2nd Class Riza Caparros from Wikimedia

Commons (left), Worradmu from FreeDigitalPhotos.net (top), and Stefan Erber (bottom)

Table of Contents

I. Introduction ......................................................................................................................... 1

II. Overview of Results Measurement in Challenge Funds .......................................................... 4

1 Articulating the Results Chains .............................................................................................. 7

1.1 Example results chains ............................................................................................................ 7

1.2 How to use results chains in challenge funds ....................................................................... 10

2 Defining Indicators of Change .............................................................................................. 12

2.1 Project specific indicators ..................................................................................................... 12

2.2 Common indicators ............................................................................................................... 14

2.3 Making projections ............................................................................................................... 16

2.4 Indicators of sustainability .................................................................................................... 17

3 Measuring Changes in Indicators ......................................................................................... 18

3.1 Working with the business to measure indicators ............................................................... 18

3.2 Self-reported information ..................................................................................................... 18

3.3 Primary data collection ......................................................................................................... 20

3.4 Additional surveys and studies ............................................................................................. 21

3.5 Collecting baseline information ............................................................................................ 22

3.6 Gender disaggregated indicators .......................................................................................... 22

4 Estimating Attributable Change ........................................................................................... 24

5 Capturing Wider Changes in the System or Market .............................................................. 26

5.1 Defining systemic change ..................................................................................................... 26

5.2 Assessing systemic change .................................................................................................... 27

6 Tracking Programme Costs .................................................................................................. 29

7 Reporting Results ................................................................................................................ 29

7.1 Acknowledge contributions .................................................................................................. 29

7.2 Responsible reporting ........................................................................................................... 29

8 Managing the System for Results Measurement .................................................................. 30

8.1 Human and financial resources............................................................................................. 30

8.2 Using results in challenge funds............................................................................................ 30

Annex A: Useful References ....................................................................................................... 32

Annex B: Systemic Change Definitions from AECF ....................................................................... 33

1

I. Introduction Private sector growth has the potential to create jobs, raise incomes, and lift people out of poverty.

Recognising this, donors are increasingly using challenge funds to support innovative business

models or projects with a potentially high pro-poor impact.

This paper provides guidance for results measurement in challenge funds using the DCED Standard,

an eight-part framework for measuring and managing results. Many of the recommendations are

also applicable to other forms of partnership with the private sector, such as loan facilities or

business support services. It could also be relevant for businesses that are interested in measuring

the results of their work. These guidelines open by presenting the DCED Standard and introducing

challenge funds. They then examine each of the eight elements of the Standard, providing practical

advice on how to implement them in a challenge fund.

There is currently little available guidance on results measurement in challenge funds. This report

draws on the experiences of several funds using the DCED Standard, as well as others that take a

similar approach. We welcome further input from practitioners and look forwards to deepening the

guidance contained here.

a) Introduction to the DCED Standard The DCED Standard provides a practical framework for private sector development programmes to

monitor their progress towards objectives. This enables them to better measure, manage, and

demonstrate results. The Standard was first developed in 2008, and has been gradually refined in

collaboration with private sector development programmes in the field. It includes eight elements:1

1. Articulating the results chain.2 Results chains visually represents how project activities are

expected to lead to outputs, outcomes and impact, showing the anticipated causal links and

relationships between them. They clearly demonstrate what the project is doing and what

changes are expected.

2. Defining indicators of change. An indicator specifies what you will measure in order to see

whether change has occurred. Indicators are specified for each step in the results chains.

3. Measuring changes in indicators. Once the indicators have been defined, they are regularly

monitored to see what has changed and to help projects manage accordingly.

4. Estimating attributable changes. Once a change is observed, you need to estimate what can be

attributed to your project. For example, an increase in jobs may be due to your project – or

because of the wider economic environment. Estimating attributable changes helps a project

identify which interventions are working and which are not.

5. Capturing wider changes in the system or market. Many private sector development

programmes aim to affect entire market systems. Monitoring these changes helps projects

identify what is working and revise implementation strategies to maximize results.

6. Tracking programme costs. In order to assess the success of the project it is necessary to know

how much was spent as well as what was achieved.

1 These guidelines are downloadable from http://enterprise-development.org/page/implementing-standard

2 Results chains are also known by a variety of other names, such as logic model, impact logic, or causal chain.

2

7. Reporting results. Findings are communicated clearly to funders, local stakeholders, and to the

wider development community where possible.

8. Managing the system for results measurement. For a monitoring and results measurement

system to be effective, it must be adequately resourced and integrated into all aspects of

project management, informing the implementation and guiding the strategy.

This guideline assumes a basic knowledge of the above framework. If you have not encountered the

DCED Standard before, review the comprehensive guidance available online.3

b) Introduction to challenge funds A challenge fund provides grants or concessional loans to projects proposed by businesses which

have the potential to solve a defined development issue.4 Funding is awarded through an open

competition, based on pre-defined criteria such as the potential for commercial viability, expected

development outcomes, and an assessment of whether the challenge fund subsidy is necessary for

that project to proceed. The business is typically required to match the donor funding, from private

investors or internal resources.

Challenge funds are used to address a wide range of development issues, including health, civil

society, environment, and private sector development. This guide concentrates on private sector

development challenge funds – often called ‘Enterprise Challenge Funds’ – which finance businesses

in order to raise incomes, provide employment, and increase access to markets for the poor. Many

lessons contained in this guide are relevant for other forms of private public partnerships.

Despite the significant funding provided to challenge funds, there is currently little evidence to show

whether they achieve the anticipated development impacts. Heinrich (2013) finds that publically

available results from challenge funds are generally anecdotal, and frequently focus on positive

stories without critically examining the impact.5 Similarly, a study by the Netherlands Ministry of

Foreign Affairs notes the ‘sparse’ evidence on the performance of public-private partnerships.6 In

particular, there is little available information on whether challenge funds can create longer-term or

systemic change. This lack of evidence poses a risk to challenge funds, which must demonstrate

impact to justify their funding. Moreover, it inhibits the ability of challenge funds to learn from their

experiences, and share this learning with others to improve their performance.

Several modern challenge funds are working to improve their results measurement, in order to

increase the evidence base and improve the quality of their work. For example, the African

Enterprise Challenge Fund (AECF) and Enterprise Challenge Fund (ECF) use the framework of the

DCED Standard, while the Food Retail Industry Challenge Fund (FRICH), Business Innovation Facility

(BIF), and the Vietnam Challenge Fund are investing in results measurement, among others.

However, there are currently no standards for results measurement in challenge funds, or guidelines

to best practice. This paper aims to addresses this gap by suggesting ways in which the DCED

3 See www.enterprise-development.org/page/measuring-and-reporting-results

4 Chilver, A., van Diermen, P., and Jones, W., 2006, Using Enterprise Challenge Funds to Promote a More

Enabling Environment for Business: Challenges and Opportunities 5 Heinrich, M., 2013, Donor Partnerships with Business for Private Sector Development: What can we Learn

from Experience, Donor Committee for Enterprise Development 6 APE., 2013, Public-Private Partnerships in Developing Countries – A systematic literature review, Ministry of

Foreign Affairs of the Netherlands

3

Standard can be used in challenge funds. It is hoped that it will contribute to strengthening our

understanding of how to effectively work with the private sector.

4

II. Overview of Results Measurement in Challenge Funds

a. Developing a ‘right touch’ results measurement system Many challenge funds use a ‘light touch’ management and measurement system, which relies

primarily on self-reported results from businesses.7 A light touch system reduces cost for both the

fund manager and the business. However, Callan and Davies (2013) note that this may be a “false

economy” if the challenge fund cannot learn from experience or demonstrate its impact.8

Fund managers must find a practical balance by investing in a results measurement system which

measures beyond outputs, validates self-reported results from businesses, and captures change in

the wider market. The DCED Standard can provide a framework to keep the system practical and

cost-effective, by preventing fund managers from reinventing the wheel or measuring unnecessary

indicators. Three key ways to reduce the cost of the results measurement system are:

Develop a clear results measurement system. Make results measurement responsibilities clear

from the beginning. Key requirements, such as reporting against indicators and hosting site

visits, should be specified in the contract with the grantee. This ensures that the business

understands the challenge fund’s expectations, and is contractually required to deliver. It is

worth investing extra resources to develop an effective results measurement system at the

beginning of the challenge fund, as changes require additional time to understand and use.

However, no results measurement system will be perfect straight away. If changes are required,

they can be piloted with a small number of capable or interested businesses. Once the revised

system is working well, it can be rolled out to other businesses in the portfolio, and integrated

into future funding rounds.

Make results measurement useful for the business. Businesses are often interested in results

measurement, for three reasons. Firstly, businesses are frequently keen to learn new ways of

keeping track of their activities and outputs. Secondly, monitoring the results of their work helps

to strengthen their own value chains, improving their understanding of their customers and

suppliers.9 For example, understanding the extent to which customers benefit from their

products can inform pricing and marketing decisions. Finally, demonstrating the impact of the

business can help build relationships with the government, attract further funding and influence

the business enabling environment. Challenge funds should consistently emphasise the

importance of results measurement for the business, and customise the system to make it as

useful as possible. This will improve buy-in from the company, and so increase the reliability of

the results reported.

Take a portfolio approach. The success of a challenge fund depends on the portfolio, not any

individual project. It is expected that some business ideas will fail, and lessons can be learnt from

failures as well as successes.10 Similarly, the challenge fund can monitor some projects in more

depth than others. Faced with resource limitations, it may be more effective to prioritise

monitoring of more expensive, successful, or innovative business projects. As the challenge fund

7 Heinrich, M., 2013, Donor Partnerships with Business for Private Sector Development: What can we Learn

from Experience, Donor Committee for Enterprise Development 8 Callan, M, and Davies, R., 2013, When business meets aid: analysing public-private partnerships for

international development, Development Policy Centre 9 WBCSD, 2013, Measuring socio-economic impact: A guide for businesses

10 Ruffer, T and Wach, E., 2013, Review of M4P Evaluation Methods and Approaches, ITAD

5

cannot identify the most successful or innovative projects straight away, they could monitor

everything to a minimum standard, and select the most interesting or successful projects for

more detailed analysis. This means that not all projects within the challenge fund will comply

with the DCED Standard, which should be agreed with the donor and transparently reflected in

the reporting. The following table gives an example of how this prioritisation may look like in

practice. However, each challenge fund will need to find its own flexible balance between what

is practical and what is ideal, so this suggestion will need to be improved and adapted for each

context.

Example prioritisation in a challenge fund:

Starting point for all projects. After eighteen months, all projects are assessed and moved into category B or A.

Results chain developed linking grant funding and ultimate impact.

Small baseline conducted after project starts but before any changes have occurred.

Indicators developed and monitored with small sample sizes.

Category B: Projects with a low size, strategic importance, potential benefits or innovation

Indicators monitored to assess attributable changes for enterprise and beneficiaries. (e.g. profits, yields)

Businesses self-report and visited to validate reports.

Beneficiary level data verified by checking with a small number during field visit.

Category A: Projects with a high size, strategic importance, potential benefits, innovation, or potential to change market system.

All ‘category B’ monitoring, and:

Indicators monitored to assess attributable change in impact indicators (such as income, other benefits for household)

Rigorous studies (mixed methods, larger samples) to validate findings and collect information from direct beneficiaries.

Systemic change assessed by talking with different players, looking at market trends, and linking to project activities.

Further case studies examine aspects of project (e.g. gender)

Closed projects: Projects

which closed because

they were not financially

successful.

If a grantee is not financially successful, monitoring could usefully extract lessons for the challenge fund and other inclusive businesses in the region. This is likely to be based around a case study which analyses the factors contributing to the closure of the project.

b. Characteristics of challenge funds The diversity of challenge funds, and lack of established best practice, mean that this guide cannot

provide definite answers for all scenarios. It gives firm guidance where possible, and highlights key

issues where not. Practitioners can use the guidance contained here to inform the development and

implementation of customised results measurement systems.

In order to help practitioners adapt advice contained here, some key characteristics of challenge

funds are described in the following table, alongside the implications for results measurement:

Characteristic of challenge fund

Implications for results measurement

What is the objective of the

Challenge funds should tailor their results measurement system to their objectives. One that aims to increase income and employment will have a very

6

challenge fund? different results measurement system to one that aims to reduce emissions of carbon dioxide. Whether the challenge fund aims to facilitate market-wide changes, and the type of change envisioned, will also affect measurement aims.

What type of company does the challenge fund work with?

Smaller companies may have less capacity for results measurement. However, smaller companies are typically more dependent on the grant than large ones, and so have a greater incentive to demonstrate results to the fund manager. Social enterprises or brand-conscious businesses may be particularly interested in measuring outcome and impact.

How focused is the challenge fund?

A geographically and sectorally focused challenge fund will find it easier to facilitate, assess, and attribute market-wide changes. It is also more practical to aggregate common indicators to understand the results of the fund. However, focused challenge funds may find it harder to monitor the impact of individual projects, given the likely overlaps in objectives and target populations between grantees.

How much money does the challenge fund disburse?

In most challenge funds, a percentage of the total budget is allocated to management and results measurement. However, the administration cost primarily varies by the number of grants disbursed, not the size of each one. Consequently, challenge funds that give small amounts of funding to a large number of grantees will have fewer resources available for results measurement than those that disburse more money to fewer companies. Similarly, a business receiving a small grant will find results measurement requirements proportionately more costly than one receiving a large grant.

7

1 Articulating the Results Chains “To a business, the results chain makes perfect sense.” – Mark Thomas, Nathan Associates

A results chain is a visual tool which

maps the activities conducted by the

business, and shows how these are

expected to lead to positive

development outcomes. It is analogous

to business tools such as a business

outcome map, which shows how

activities lead to business

improvements.11

Results chains have three main benefits.

Firstly, they clarify the objectives and

logic of the project, showing why the

fund manager believes that each grant

will have a positive impact, and which

assumptions this relies on. Secondly,

they show what the business hopes to

achieve out of the partnership (such as

increased profit and turnover), what the

fund manager expects (development

outcomes, systemic change), and the

linkages between the two. This helps

establish a joint understanding of the project between the fund manager and the business. Finally,

results chains provide the basis for results measurement, by clarifying what needs to be measured in

order to see to what extent the project is achieving its expected results.

This section first provides examples of results chains and then gives advice on how to use results

chains in challenge funds. It assumes the readers are familiar with the concept, and so does not

provide general guidance or principles. If you have not worked with results chains before, download

the DCED Guide to Results Chains for further explanation.12

1.1 Example results chains

1.1.1 Beneficiary as supplier

Challenge funds frequently finance businesses where the beneficiary is expected to profit by

supplying the company with a product. For example, farmers may sell directly to a business, or may

sell to intermediaries in the value chain. In the below example from ECF, the business buys abaca

fibre directly from farmers. It expects to benefit directly employed workers (shown on the left),

farmers selling abaca fibre (in the middle), and anticipates wider systemic impacts (on the right.)

11

WBCSD, 2013, Measuring socio-economic impact: A guide for businesses. For more on business outcome maps, see Programme and Business Value Management, A brief description of Business Outcome Maps. 12

Available at www.enterprise-development.org/download.aspx?id=1833

Money doesn’t buy happiness

Financial sustainability is no guarantee that the project

is having a positive pro-poor impact. For example, the

business might profit by:

Supplying to or purchasing from wealthier citizens.

Supplying services which have negative, longer-

term effects. For example, a business might

displace existing farmers when they purchase land

for a factory.

Working primarily with intermediaries who do not

pass on additional income to the poor.

Displacing competitors who do not have a

subsidised loan, but not offering a substantially

improved service.

By illustrating the links between the business activities

and development outcomes, developing the results

chain should show why the challenge fund expects the

project to be beneficial for the poor.

8

9

1.1.2 Beneficiary as customer

In other businesses, the beneficiary is expected to profit from purchasing from the company. This

may require a more complex results chain, as there are multiple ways in which the customer may

benefit. These may include:

Increased access. Challenge funds may increase access to key inputs, such as improved seed,

fertiliser, or financial services, which the consumer can use to increase their income.

Better quality produce. For example, a pesticide company may develop a new type of pesticide,

which enables customers to grow a higher yield of rice.

Lower costs. For example, a solar lighting company may market a solar lamp that saves the

customer money on kerosene and charcoal.

In the below example from AECF, a company (‘Suppliers Inc’) supplies animal health products to a

rural community, and expects this to increase incomes for farmers and create jobs directly.

Anticipated systemic change is shown at the top of the results chain.13

13

The name of the company has been changed, and all figures removed to protect confidentiality. It has also been slightly simplified for the purposes of illustration.

XX households to benefit by $XX a year by 2016

XXFTE jobs by 2013, Annual wage bill of USDXXX by 2016

Project turnover and profit

50 stores operational by 2016 offering animal health advice and Suppliers Inc products

Farmers are more knowledgeable about animal husbandry

Suppliers Inc set up distribution network for their products

Grant $XXX Repayable grant $XXX Own company funding $XXX Inputs

Activities

Outputs

Business Performance

Development Performance

Farmers buy animal health products from stores

Farmers get cheaper animal health products

Suppliers Inc train farmers in animal husbandry

Suppliers Inc train distribution network staff in basic animal husbandry

Store staff provide health advice to farmers

Farmers travel less far for information and health products

Farmers have healthier animals

Farmers use new knowledge and materials.

Systemic Change

Selling animal health projects to smallholder farmers is seen as a viable business model, and more companies enter the market.

10

1.2 How to use results chains in challenge funds Develop results chains early. Develop results chains early in order to inform the design of the

project results measurement system. One approach is to develop results chains for each project

before the contract has been signed, but after funding has been awarded. This allows indicators

to be derived based on the results chain and included in the contract. With sufficient resources,

it may be valuable to develop results chains for short-listed applicants before funding is

awarded. This would allow results chains to inform investment decisions, for example by

suggesting that a profitable business is unlikely to deliver pro-poor outcomes. However, it also

entails additional costs during the competition stage, and so may not always be practical.

Keep it simple. The results chain should be kept clear and logical so that it is easy to understand

how activities lead to expected changes in business and beneficiary behaviour. The DCED

Standard says that the results chain should express ‘key changes’ in the project. In practice, this

requires a certain amount of judgement; the example results chains in the previous section will

give an indication. As a rough guide, if you can’t print it out in an easily readable size on a single

page of A4, then it is probably too big.

Results chains are the responsibility of the fund manager. The person who manages the grant

should manage the development of the results chains, to ensure that they can use the collected

information to inform project oversight. While a results measurement specialist can support this

process by providing training or quality assurance, they should not have overall responsibility for

the results chains. Some challenge funds go further and ask the business to develop and manage

the results chain; see the experience of FRICH, below.

Involve the business during the development and revision of results chains. In all cases, the

results chains must be developed in collaboration with key stakeholders from the business. This

typically would include the project manager, CEO, and field staff. Businesses typically enjoy

discussing the logic of their project, and find it helpful in sequencing their activities and thinking

through expected behavioural changes from their customers and suppliers.

Focus on key changes in beneficiary behaviour. The results chains must specify expected

intermediate changes in the attitudes and behaviour of the ultimate beneficiaries, not just that

of the business. These links and assumptions should be supported by background information,

which justifies the expectation that the business will lead to benefits for its suppliers or

customers. This information will typically be gathered during the competition stage, and inform

the decision to award the grant. These links and assumptions should also be checked through

Requiring businesses to develop results chains – the experience of FRICH

The Food Retail Industry Challenge Fund (FRICH) requires grantees to develop and own the

results chain. This encourages the business to think about the outcomes of their work, improves

ownership of the results measurement system, and improves the prospect for the emergence of

a sustainable business model.

FRICH work primarily with large UK retailers, who have the time and capacity for results chain

development. The DCED does not recommend this approach for all challenge funds, as it may

not be practical for smaller grantees with less time or resources. However, it is a valuable model

and we encourage other challenge funds to experiment with ways to further involve the

business in results measurement.

11

regular field trips, which must involve discussions with the end beneficiaries to validate these

assumptions and check that they are realistic.

Keep results chains flexible. Companies adapt their business plan over time, which can render

the original results chain outdated. The challenge fund must schedule in regular revisions to the

results chain (at least annually), and allow businesses the flexibility to change activities within

the scope of the project objectives. Use the results chain discussion to establish whether

changes in strategy have affected the outcomes. For example, if a business switches suppliers

from smallholder to commercial farmers, it may no longer have a pro-poor impact.

12

2 Defining Indicators of Change “The question is not how many indicators you have, but how good they are.” –Wafa Hafiz, consultant

An indicator shows what change you expect to see at each level in the results chain, and so what you

will measure in order to check whether the anticipated outputs and outcomes have occurred. If you

are unfamiliar with indicators, download the DCED Guide to Developing Indicators.14

2.1 Project specific indicators The DCED Standard requires every output and outcome in each project results chain to have an

indicator. For example:

Results Chain Indicator for Each Level

Income from additional crop sold.

Amount of additional crop sold by farmer.

Amount of yield lost due to crop pests.

Number of farmers using bait. Ways that farmers use bait (qualitative).

Amount of fruit fly bait sold. Number of farmers purchasing fruit fly.

Numbers of farmers trained.

Indicators linked to a well-designed results chain provide valuable information for both the grantee

and fund manager. The process of defining them based on the results chain clearly shows the

business why they are relevant to the project, and so why they need to be monitored.

Many of the indicators are directly relevant to business performance, especially at the lower and

middle levels. In the above example, from the Enterprise Challenge Fund, the business aimed to

increase crop yield by introducing a new type of fruit fly bait. Monitoring revealed that, although the

farmers attended the training and understood how to use the bait, they were not purchasing or

using it. Further investigation showed that farmers were concerned that, if the bait did not work as

promised, they would be left with unmanageable debt. This monitoring helped the business to

identify and address these issues. The challenge fund should thus encourage the grantee to regularly

monitor key indicators and use the information to manage their business. While some grantees will

do this anyway, others – especially smaller, less experienced businesses – will benefit from the

support of the fund manager.

14

Available at www.enterprise-development.org/page/download?id=2132

Farmers increase income

Farmers sell additional crop

Reduced loss from crop pests

Farmers use organic fruit fly bait correctly

Farmers purchase organic fruit fly bait

Business provides training on using organic fruit fly bait

13

At a higher level in the results chain, indicators show the extent to which assumptions about how

the business benefits poor people are holding true. This helps the challenge fund manager learn

what types of businesses and business models are more likely to result in significant impacts, and

can inform future fund management. In many cases, though not all, this information is also relevant

for the business. In the above example, monitoring the effect of the bait on income will help with

pricing, marketing, and accessing further finance. However, businesses will often not have the

capacity or interest to monitor high level indicators, as discussed further in chapter three, Measuring

Changes in Indicators.

Measuring changes throughout the results chain also helps demonstrate that the observed changes

are likely due to the project. They build a ‘credible story’ that links project activities with outputs,

intermediate outcomes, and final outcomes. This is discussed further in the DCED Guide to

Attribution.

Results Measurement – An Alien Concept?

It can be challenging to engage businesses in results measurement. The Business Innovation Fund (BIF) gives relatively small amounts of technical support to each business, and so cannot easily enforce reporting against indicators. Consequently, it strives to make results measurement useful for the business, in order to encourage reporting. This starts with a half-day introductory workshop, during which indicators are agreed and baseline information collected. Simple, participatory methodologies are used to structure the session. For example, at the beginning of each workshop grantees are asked to draw what an alien would see if they visited the project after three years – a visual way to clarify the planned impact. (Pictured above). This impact becomes the end-point of a basic results chain, developed using pieces of card pinned on wall. The results chain takes just twenty minutes to make, but encourages the grantee to think about the desired outputs and outcomes of the programme. The results chain becomes the basis for developing grantee-specific indicators. It ensures that the indicators are closely tied to the logic of the project, and shows the grantee why they are worth measuring. It provides the opportunity for BIF to suggest indicators and ways of measuring them. For example, if a seed supplier wants to measure the number of farmers they reach, they will typically need to design a way of tracking their customers. BIF suggests ways in which this can be done, and reasons why it is important.

2.1.1 Managing indicators

Challenge funds often have limited capacity for additional measurement, and so must base their

results measurement system on a continuing assessment of what is possible. This section proposes

three strategies to ensure that you are not trying to measure too much. See chapter three below for

advice on how to simplify the monitoring of each indicator.

Photo credit - Caroline Ashley

14

Understand what the company can provide. Define the indicators in collaboration with the

grantee, in order to understand what information they can provide. This will avoid a ‘wish-list’ of

indicators, and allow you to carefully consider what additional monitoring is needed. Be aware

that the company will not be able to spend significant amounts of time collecting additional

information, beyond what is in their interests to measure.

Reduce number of indicators. If necessary, cut the number of indicators that you wish to

measure. Pick one good indicator for each step in the results chain, rather than several. While

the DCED Standard requires every step in the results chain to have an indicator, this may not be

practical in all cases, If you are not measuring steps in the results chain, then draft a brief note to

explain why not. If you are not measuring a change, then you cannot claim it as an observed

impact of the project.

Gather qualitative information. Indicators typically gather quantitative information, which is an

essential part of a results measurement system. However, challenge funds must also gather

qualitative information, which tell the story of why indicators are changing. In the above

example of a company selling a new type of fruit fly bait, you would explore how farmers use the

bait, the extent to which they are satisfied, and whether they believe it is effective.

2.2 Common indicators In addition to project-level indicators, all challenge funds should define a small set of common

indicators, which all (or most) projects use. This allows results to be aggregated, compared, and

reported against. Challenge funds are recommended to only use a small number of common

indicators, as they can be complicated and time-consuming to measure.

2.2.1 Common outcome indicators

The DCED Standard recommends the use of three universal indicators; scale, net income, and

additional jobs created.15 While these are applicable to a variety of private sector development

programmes, they may not be relevant for all challenge funds, which often have a wide range of

expected benefits.

Consequently, common outcome indicators should be carefully selected based on the purpose of the

challenge fund, and measure a factor which the challenge fund expects to influence and change.

In a diverse challenge fund, not all projects may be able to report against all common indicators. For

example, challenge funds might wish to measure additional income for households participating in

the projects. However, a business that provides a banking service, allowing people without access to

formal banking service to save money through their mobile phone, may be unable to track the

additional net income that results. It may be too complex to track the numerous links between

financial services and income, or a change in income may not be a primary benefit. In this case, it is

important to accept these limitations, and find alternative indicators that better track whether the

project is beneficial.

15

DCED, 2013, The DCED Standard for Measuring Achievements in Private Sector Development, Version VI

15

2.2.2 Common financial indicators

Key financial indicators, such as profit and turnover of the project supported, are a crucial measure

of success. By indicating whether the project is financially viable, they help understand whether the

project is likely to be sustained or replicated.

Measuring profit and turnover is relatively simple if the grantee is a new start up, or the grant is

funding the whole business, as profit and turnover can be derived straight from the company

accounts. It is important to use a standard, uniform measure of profitability to enable comparisons

within the portfolio. For example, AECF measures profit using EBITDA; earnings before interest, tax

and depreciation.

It is more complex if the grant is funding just one part or project of a larger business. In this case, it is

possible to require businesses to develop a separate profit and loss account for the project on their

accounting system, so that they can develop expenditure and revenue lines which are specific to the

challenge fund project.

This can be challenging for the business, and it may require them to invest in specialised accounting

software and a professional accountant. AECF finds that smaller businesses are typically eager to

acquire these skills. ECF paired one of their partners with an Australian Business Volunteer, who was

able to improve reporting on the grant while also boosting overall financial management capability.

2.2.3 The challenge of aggregation

‘Aggregation’ is the technique of combining data on one indicator from several projects, in order to

provide an overall understanding of a portfolio. For example, if ten projects in a portfolio each

create 20 jobs, an aggregated indicator would show that the portfolio created 200 jobs. Aggregation

Is GIIN a Tonic?

The Global Impact Investing Network (GIIN) has developed a set of common indicators, known as

IRIS, designed to measure social, environmental, and financial performance, evaluate deals, and

increase the credibility of the impact investing industry.. They aim to create a common language

for reporting social and environmental performance, enabling fair comparisons between

different projects, benchmarking performance, and communicating impact. Almost all are output

indicators, measuring the direct results of the business.

The DCED Standard and the IRIS indicators are complementary. The DCED Standard provides a

framework for the development of a results measurement system, while the IRIS catalogue

provides well-defined and widely-established indicators.

Challenge fund managers can use IRIS to ensure that they are consistent with other impact

investors, and as a source of inspiration when selecting indicators. They are particularly useful for

setting common indicators, as each indicator has a clear definition designed to allow for

aggregation. The use of the IRIS indicators may also help the business attract other impact

investment capital.

More information on the IRIS indicators can be found online at http://iris.thegiin.org.

16

is a valuable but challenging technique, and requires a significant investment in time and rigour in

order to provide meaningful analysis. In particular:

Clearly and consistently define indicators. If indicators are given different definitions in

different contexts, then it will not be meaningful to aggregate them. For example, suppose that

one company considers short term, seasonal agricultural labour to be a ‘job’, while another

company counts only full time jobs. A comparison between the jobs created by these two

companies will not tell you anything useful; any genuine difference in job-creation will be

masked by the difference in definitions. Consequently, it is vital to ensure that indicators are

consistently defined, and this is clearly communicated to the companies or staff who measure

them. The IRIS indicators (described above) contain standardised definitions of indicators,

designed to promote comparison between businesses.

Consider context when making comparisons. The more similar projects are, the more

meaningful it is to aggregate indicators. For example, a flower business in Kenya may be much

more financially successful than one in Sierra Leone, as there is already a thriving flower industry

in Kenya. However, a moderately profitable project in Sierra Leone may represent a better

investment than a very profitable one in Kenya, given the difficulties of conducting business in

that environment. Consequently, when aggregating and making comparisons it is essential to

take context into account.

Avoid overlap. A geographically concentrated portfolio may have significant overlap between

different projects. In this case, the overlap should be removed to avoid double-counting. For

example, if one project supplied fertiliser to 200 farmers, and another project purchased maize

from the same farmers, a simple aggregation of the two projects would conclude that the

portfolio reached 400 farmers. In reality, however, it would just have been 200 farmers.

2.3 Making projections Projections should be made for key quantitative indicators throughout the lifetime of the grant, and

even beyond if beneficiary level impact is expected after the full grant has been disbursed.

If these projections are made during the application stage, they can also be used to select winning

proposals. For example, businesses might be asked to predict the number of households reached, or

their impact on income. In this case, businesses tend to give extremely optimistic projections, in the

hope of securing funding. (They are much like NGOs or consulting firms in that respect.) The fund

manager will need to discuss with businesses carefully before using projections as a basis for

awarding funding, and often revise them down before they can become realistic targets.

Projections also enable the fund manager to hold the business accountable for deliverables, and

assess project progress. In some challenge funds, funding is disbursed when the grantee meets

agreed targets. In this case, it would be particularly important to independently verify reported

results.

Be aware that business projects frequently change during implementation. Consequently, early

projections can soon become outdated. The results measurement system should include

opportunities to revise projections at least annually.

17

2.4 Indicators of sustainability The primary indicator of sustainability in a challenge fund is the commercial viability of the business.

If it is profitable or likely to be so without the support of the challenge fund, then the project is likely

to be sustained. This information should be supplemented by qualitative information that explores:

Do key decision makers in the company back the continuation of the project?

Does the company have a clear plan for the continuation or expansion of the project?

Is the company able to access finance for the continuation or expansion of the project?

Is the company investing further resources into the project, without public support?

Are there any risks to the business model in the near future?

Does the business model have any long-term adverse effects?

Are all actors along the supply chain of the company profitable, and willing to continue

interacting with the company? For example, a business which supplies fertiliser to farmers is

only sustainable if the farmers benefits from the use of the fertiliser. Similarly, a business which

purchases from smallholder farmers is only sustainable if the farmers wish to continue working

with the company. In many companies, consequently, a high ‘development impact’ can indicate

sustainability.

As well as assessing financial sustainability, it would be valuable to assess the sustainability of the

development outcomes, by exploring:

Does the business model have any long-term adverse effects?

Do the continuation plans of the company involve any changes to the business model that might

affect the benefit for the customers/suppliers? For example, there might be a change in planned

levels of employment, or a reduced focus on working directly with small-scale farmers.

18

3 Measuring Changes in Indicators “Tracking results can help you, the management, to stay focused and adapt quickly.” – Innovation

Against Poverty, Guidance for Applicants.16

Measuring changes in indicators allows you to monitor progress, report on successes, and improve

less effective areas. This section first describes how responsibility for measuring indicators must be

agreed between the fund manager and the business. It then suggests ways for the fund manager to

cost-effectively gather information, and collect baseline data.

Additional information can be found in the relevant DCED Guide to Measuring Changes in Indicators

and the DCED Guide to Good Research Practices.17

3.1 Working with the business to measure indicators Challenge funds must divide up responsibility for measuring indicators between the fund manager

and the business. Roughly speaking, the business should report on indicators that they collect as part

of their everyday work, and which do not require extensive additional data collection. These are

typically activities and outputs, although a small number of businesses may also track outcomes. The

fund manager should lead on additional data collection, in particular related to development or

systemic impact.

3.2 Self-reported information The results measurement system starts with information reported by grantees. Businesses typically

regularly report on:

16

Innovations Against Poverty, Note for Applicants for IAP Financial Assistance 17

DCED Guide to Measuring Indicators available at www.enterprise-development.org/page/download?id=2111, and DCED Guide to Good Research Practices available at www.enterprise-development.org/page/download?id=2133

The Last Hard Number

AECF have developed the useful concept of the ‘last hard number’, to help them measure their

indicators. The last hard number is the furthest point up the results chain that the business is able to

measure, without recording any additional data. Once you know this, you will know what additional

information the fund manager needs to collect in order to assess progress against higher level

outcomes.

For example, an outgrower scheme is likely to be able to tell you how many farmers it worked with,

how much money it paid to each, and often what the farmers’ costs are. This means that the ‘last

hard number’ is actually extremely close to the net benefit per household. Additional surveys should

concentrate on establishing the farmers’ opportunity cost; the amount which they could have

earned if they had not joined the outgrower scheme.

A supplier project, by contrast, may know how much produce it sold – but not how many people

bought it, how they used it, or what benefit they derived from it. This means that their ‘last hard

number’ is some distance from the final development impact, and more primary or secondary

research is likely to be needed.

19

Activities against work plan.

Outputs produced.

Business performance indicators (e.g. profit, turnover).

Finances (amount spent, budget vs. actual).

Lessons learned.

The below table suggests common indicators that different types of company are able to easily

report against.

Type of business Indicators that they can often easily report against

Outgrower scheme – contracting farmers to provide produce at a fixed price

How many farmers they work with.

How much produce they bought, and how much they paid.

If the business also provides inputs (such as fertiliser or pesticide) then they will also know how much they charged the farmer for this service.

Supplier of new improved seed Quantity of seed sold.

Likely improvement in yield (based on secondary research)

Cost of seed to farmers.

Financial service provider (e.g. mobile money).

Number of service users.

Number of active users

Volume/quantity of use (e.g. number of transactions).

Cost of service to users

However, there are a number of limitations in self-reported information. Firstly, it can be difficult to

gather, as companies may see reporting as an additional burden. Moreover, reporting may not be

entirely accurate. This is particularly true if future grant disbursements rely on past performance, as

the company has an incentive to present a positive view. Consequently, challenge funds which

disburse based on agreed milestones must take particular care to verify the results reported by

grantees.

To address these issues, challenge funds can:

Use simple, clear and consistent reporting templates. This helps the business get used to an

unfamiliar requirement. Minimising the use of development jargon makes reporting more

accessible to the business, which improves quality and compliance. For example, asking about

‘systemic change’ is unlikely to get a good response. The reporting requirements should be

written into contractual agreements, so that the company is aware from the beginning of what

will be expected from them.

Support the grantee. If businesses are required to report on indicators at higher levels in the

results chain (such as the net increase in income), then they will need a lot of support. In

practice, it is likely to be a joint effort between the business and the fund manager. One useful

approach is to develop a clear model linking the indicators that the business supplies (the ‘last

hard number’, above) with the final outcomes, using clear assumptions and calculation to show

how the business’ indicators translate into outcomes. Moreover, the fund manager should work

closely with the grantee to ensure that common indicators, such as jobs created, are measured

consistently to enable comparison between different businesses.

20

Incentivise good reporting. There are a number of ways to incentivise good reporting. Most

obviously, make disbursements conditional on the reception of high-quality reports. After the

grant has been exhausted, there will be no financial incentive for the company, but reporting

may be still required to show long term impact. In this case, reduce the frequency of reporting,

and consider offering publicity to companies with strong development impacts.

Build a good relationship. The business should feel able to report failures as well as successes. A

good personal relationship with the grantee, whereby they feel comfortable discussing and

learning from their experiences, is essential.

3.3 Primary data collection

3.3.1 Business visits

The fund manager should visit the business on a regular basis, in order to review key results

measurement documents such as the results chain and indicators, collect additional information

against indicators, and check that self-reported information is accurate. It can also be an opportunity

to offer technical assistance to the business, or build their capacity in reporting.

Site visits (along with other results measurement activities) can come at a considerable cost in staff

time and resources to the business. Try to minimise this by keeping site visits short – it is not

recommended to spend more than two days directly with the grantee, although this varies with the

type of project, size of grant, and regularity of visiting. One useful way to minimise the burden for

the business is for the challenge fund to assign a single staff member to manage all contact with the

grantee, and go on each site visit. This builds up a relationship between this staff member and the

grantee, and ensures that the business does not need to repeat information for new visitors.

In order to reduce the cost to the grantee, emphasise that it is not necessary for the CEO of the

business to spend all their time with you, although they would typically be needed for key

discussions. Junior staff can take at least some of the responsibility, and field visits can be managed

primarily by extension workers.

3.3.2 Field visits

In order to assess the outcomes and impact of the project, the fund manager will typically need to

visit the field and speak with primary beneficiaries and other partners. This can verify information

from grantee reports, and collect additional information which the business is not able to provide.

An effective field visit can use a relatively small sample size. Speaking with 10-20 people is enough to

get an indication of the progress of the project. This can be supplemented by occasional larger

studies (see below), especially for priority projects. The minimum sample size will however vary by

Assessing business interest in results measurement

All challenge funds should consider how much the company cares about results measurement. In

some cases – particularly social enterprises or brand-conscious businesses – the company may

take a great interest, be eager to learn and willing to put their own resources into results

measurement. In other cases, the management will not see it as a concern. This must be taken

into account by the challenge fund – perhaps by asking enthusiastic companies to take on more

results measurement responsibilities or prioritising impact assessment for those companies.

21

project; for example, a more heterogeneous group will need a larger sample size. One way to

conduct these surveys is to partner with a local research firm, who can make staff available at a

relatively low cost on a flexible basis. Alternatively, hire and train in-house staff who speak the

language and can travel out to the field.

Be aware of the biases inherent in field research. Try not just to visit customers and suppliers

suggested by the business, which is likely to select the best cases. Instead select randomly, or visit a

central place (such as a village market) where you can find people to interview. The staff of the

business will normally be needed to make introductions and show you round, but ask them not to

join the interviews, as they will inhibit the respondent from speaking freely. For the same reason, it

is better to travel with local, junior staff rather than senior managers. Finally, ensure you speak with

a mix of old, young, male and female respondents.

3.4 Additional surveys and studies Additional studies will usually need to be carried out if the project aims to fully understand the

impact. They can survey larger numbers of beneficiaries than is possible during brief field visits, and

gather information on the higher levels of the results chain. Additional studies are typically primarily

managed by the fund manager, unless the business is particularly interested in establishing their

impact. However, businesses should input into the design, as detailed in the above box.

Surveys can be outsourced to external consultants, or conducted in-house. The advantage of the

former is that it can ensure extra rigour, especially if the subject requires specialist expertise. For

example, ECF commissioned the Micro-insurance Learning and Knowledge Centre to study the

benefits of a micro-insurance product they funded.18 The Business Innovation Facility are partnering

with the Institute of Development Studies, who designed a case study methodology to be conducted

by MBA students from the Said Business School of Oxford University, for eight of the most

interesting companies.

In-house studies can reduce costs and ensures that results are understood and used by the fund

manager. Even if outsourced, the fund manager will need resources in order to ensure that the

methodology and design are appropriate, closely supervise the research and understand and check

the quality of the final product.

18

Budzyna, L and Chandani, T., 2013, “Doing the Math” – Loan protection insurance in Cambodia, Microinsurance Learning and Knowledge Centre

Making Results Measurement Useful for the Business

Fund managers should ensure that additional data collection is useful to the business. There will

not be a complete overlap, but the business will benefit from better understanding their

customers and suppliers. Consequently, invite the business to input into terms of reference,

questionnaires, and survey design, share information with them, and provide examples of how

other businesses have used this to improve their business. For example, AECF asks companies to

input into any studies and surveys which are being conducted. One company asked AECF to

include questions about the willingness of their suppliers to pay through mobile phone –

gathering valuable information to inform their business model.

22

3.5 Collecting baseline information Any indicator that aims to demonstrate change needs to have a baseline, which shows the situation

at the beginning of the intervention and allows comparisons to be made to the situation at the end.

The business can provide baseline information on indicators which they subsequently report against.

For other indicators, the fund manager will have to conduct separate data collection. This is often

difficult to do at the beginning of the project, as the company may not have selected their

intervention area or customers/suppliers. Consequently, investing in a large baseline survey at the

outset can be ineffective, as a change in the business can render the information irrelevant.19

Moreover, conducting a large baseline for every grantee may be prohibitively expensive – and you

will not know which grantees merit closer monitoring until some time has elapsed. While there is no

perfect solution, a challenge fund can adopt the following options:

Plan the baseline in advance. Aim to conduct the baseline after the target group has been

identified, but before changes are expected to occur. This can be challenging, especially in an

agricultural project where timings are affected by the season and so may be inflexible.

Consequently, consider developing the terms of reference, advertising, and contracting

consultants (or arranging internal resources) for the baseline before the exact date has been

selected. This will give you the flexibility to quickly conduct the baseline once the company has

chosen its customers or suppliers.

Conduct a small initial baseline. An initial sample of 10-20 people can be enough to gather basic

baseline data in the majority of projects. Although this will not provide statistical rigour, it will

provide key information on the indicators, and limits the waste of resources, should the business

plan change. It will also provide crucial initial information to see how successful the project is,

which can be followed up by larger studies at a later date.

Expand the baseline as necessary. For those companies that show evidence of strong

development impacts, a more rigorous study can then be planned and conducted. Successful

companies usually expand so baseline information can be gathered from new suppliers or

customers that are part of the expansion plan. Impact can be assessed after these people can be

expected to have benefited.

3.6 Gender disaggregated indicators The DCED Standard requires key indicators to be disaggregated by gender, a proxy for the project’s

impact on gender equality.

In a challenge fund, the starting point must always be the information that is already collected by

the company. Consequently, the first step will be to meet with the company to discuss what data

they already collect, and what they can easily share with you. For example:

Employment. Businesses can typically provide gender-disaggregated data on their direct

employees.

Contracted suppliers. If a business contracts small farmers to supply them with produce, they

will typically be able to provide gender-disaggregated data on their suppliers, as they will have

information on the person who they signed the contract with.

19

Ruffer, T and Wach, E., 2013, Review of M4P Evaluation Methods and Approaches, ITAD

23

Purchasers. A business is unlikely to be able to provide gender disaggregated data on their

purchasers. For example, a seed seller would not know if the purchasers were male or female.

If the business cannot report on gender indicators, then the fund manager should collect this

information on field visits, additional surveys or from secondary sources. If it is not possible, then

write a brief note explaining which indicators are not disaggregated by gender, and why.

Use the opportunity to highlight the business case for monitoring gender. One study by ECF into

women’s economic empowerment noted that “highlighting industry inefficiency and underuse of

50% of potential customers or suppliers tended to resonate with otherwise sceptical companies.”20

This prompts grantees to consider their impact on gender, which could have wider benefits beyond

the project alone. Gender disaggregated data can demonstrate to companies their gender

imbalance, which might prompt them to consider changes.

Gender disaggregated indicators – like any indicator – will not carry much information without

qualitative information. For example, if 70% of purchasers of improved seed are female, this may

indicate that more women benefit from the project – or simply that women are more likely to do the

shopping. Another common limitation is that gender-disaggregated indicators tend to focus on the

economic status of participants, which is just one aspect of women’s economic empowerment.

Although the DCED Standard does not require projects to monitor beyond disaggregated indicators,

it is good practice which has the potential to substantially improve programmes. It is also a key focus

for many donors. For more resources on monitoring gender in private sector development, consult

the M4P Hub Guidelines for Incorporating Women’s Economic Empowerment, and the Enterprise

Challenge Fund’s paper on Women’s Economic Empowerment – Practice and Policy Implications.

20 Nethercott, K., Jago-Bassingthwaighte, M., and Jupp, A., 2013, Women’s Economic Empowerment – Practice

and Policy Implications from the Enterprise Challenge Fund, Coffey International

24

4 Estimating Attributable Change Wherever a project is implemented, many changes will occur over time. Some of these changes may

be unrelated to the project, and would have happened regardless of whether the project was ever

funded. Other changes occur as a result of the project, and these changes can be attributed to the

project.

Challenge funds need to address two key issues of attribution:21

Input Additionality: This is the question of whether the grantee would have made the

investment without funding from the challenge fund. For example, the grantee might have been

able to access commercial funding, or put more of their own resources into the project. In other

words, are the financial inputs provided ‘additional’ to existing sources of funding?

Project-level attribution: This is the question of whether the observed benefits would have

occurred without the project. For example, an increase in income for farmers may be due to a

change in crop prices, rainfall, or economic growth – not necessarily related to the business.

How do you know to what extent the observed outcomes were due to the grantee?

These two questions will not necessarily have the same answer. It is possible to have a project with

high additionality, but low project-level attribution. This means that the project would not have

been implemented without challenge funding – but didn’t really help anyone. By contrast, a project

may have low additionality but high project-level attribution; it did not need challenge funding to

proceed, but did make a real difference to the lives of its participants.

Consequently, it is essential for challenge funds to address both issues in order to be confident that

their work is having the desired impact. Addressing attribution will typically be the primary

responsibility of the fund manager, although businesses will provide essential information to inform

the assessment.

Input additionality is typically assessed ex ante, before the intervention is started, and based on a

strong knowledge of the context and local business environment. Consequently, it is lies outside the

scope of these guidelines, which focus on ex post assessment, during and after the intervention has

started. This DCED hopes to address additionality in a proposed separate guidance note.

Assessing project level attribution is not fundamentally different in challenge funds than other

private sector development interventions. Indeed, because challenge funds intervene directly in the

market and have a clearly identifiable set of beneficiaries, change may be easier to attribute than in

other private sector development programmes. More information on assessing project-level

attribution, including both quantitative and qualitative methods, is available in the DCED Guide to

Attribution and the second chapter of the DCED Guide to Research Practices. The World Bank also

provides a more technical (but still accessible) resource on attribution available here.22

21

Multiple terminology is used discuss attributions. Some use ‘additionality’ to refer to attribution more generally, others break additionality down into multiple components. For a useful discussion, see Heinrich (2013). 22

DCED Guide to Attribution available at www.enterprise-development.org/download.aspx?id=2012, to research practices at www.enterprise-development.org/page/download?id=2133.

25

Evaluations of challenge funds will also seek to establish the contribution that the challenge fund has

made to higher level change in the broader market systems. One approach for impact evaluation of

private sector development programmes is to combine project level evaluation (a ‘bottom-up’

approach) with a sector-wide assessment of the broader changes caused, linking the two with a clear

theory of change.23

23

ITAD, 2013, Gems Results Measurement Handbook

Attribution in practice

Attribution is often assessed by examining appropriate control groups, to measure the difference in

results between those who participated in the intervention and those who did not.

In challenge funds, it is often complex to find appropriate control groups. This is because those who

participate in the intervention are generally qualitatively different from those who don’t. For

example, farmers who purchase improved seed may be more wealthy, enterprising or educated than

those who don’t. Similarly, companies may select the most hard-working farmers to supply them with

produce. Consequently, differences between the two groups may not be due to the intervention.

One potential approach is to compare participants who entered the programme in different years –

on the assumption that they are broadly equivalent to start with. For example, you could compare

farmers who joined in year one, with those who joined in year two. If combined with a ‘difference-in-

difference’ methodology, this may enable a valid assessment of attribution. It may not be perfect, as

for example variations in weather between the seasons will affect yields; nonetheless, it may be the

most practical method.

Alternatively, programmes may conduct in-depth case studies which follow the same people over a

period of time. If conducted rigorously, these can often disentangle the factors leading to changes in

their circumstances, and so will point to the impact of the programme.

Overall, there are many methods for assessing attribution, as described further in the documents

cited above. There is no one perfect method, and the DCED Standard does not require complete

certainty. Instead, it requires that programmes tell a story that would convince a reasonable (but

sceptical) outsider. Carefully considering the methods used, and tracing attribution through different

stages in the results chain, will help you to do this.

26

5 Capturing Wider Changes in the System or Market “This is the hard part!” – Buddhika Samarasinghe, Nathan Associates

Systemic change refers to the impact of the challenge fund on the systems that the businesses work

in, such as markets, government, or civil society. Systemic change can have a greater impact than

direct assistance, as it will benefit people who have had no contact with the programme. It is more

likely to be sustainable, because it depends on multiple actors in the system, rather than individual

companies.

There is little consensus on whether challenge funds are an effective mechanism for facilitating

systemic change.24 A lack of evidence on this issue points to the importance of newer challenge

funds effectively assessing their contribution.

One emerging lesson is that systemic change is more achievable if the challenge fund portfolio is

concentrated geographically and thematically.25 Several successful businesses in the same sector or

area are more likely to encourage new entrants, or encourage an increase in private finance. As a

consequence, significant systemic change is likely to be caused by the portfolio as a whole, rather

than a single project.

The DCED has published a guide to assessing systemic change, available here.26

5.1 Defining systemic change Systemic change is discussed more often than it is defined. It is crucial that each fund manager

clearly understands exactly what they mean by ‘systemic change’, in order to be able to assess the

extent to which it occurs or not.

For example, AECF have defined six categories of systemic change, provided in full in Annex B. These

guide project managers when conducting site visits, suggesting what they should look for and what

questions they should ask. This categorisation is also used to code and analyse the qualitative

information that AECF receives, which shows the most common types systemic change across the

portfolio.

The design and objectives of the challenge fund affects the type of systemic change that is expected.

For example, FRICH develops linkages between UK retailers and small-scale farmers in Africa. Since it

works with purchasers with significant market share, one key potential systemic change is the

replication of innovation within the grantee. This means that the grantees better appreciate the

value of working with small-scale African farmers, and expand the model to other parts of their

business. This would not be a priority for challenge funds working with small local businesses, who

might be more interested in the replication of the business model by other companies.

24

See Elliot, D., 2013, Exploding the Myth of Challenge Funds – a start at least…, Springfield Centre and Chilver, A., van Diermen, P., and Jones, W., 2006, Using Enterprise Challenge Funds to Promote a More Enabling Environment for Business: Challenges and Opportunities for two perspectives on this. 25

Poulton, C., 2009, An Assessment of Alternative Mechanisms for Leveraging Private Sector Involvement in Poorly Functioning Value Chains, FAO 26

Available at www.enterprise-development.org/page/download?id=2113

27

Systemic change should be viewed broadly, as a successful business may inspire change not just

within the sector, but in other sectors or countries. For example, a company that successfully

engaged with consumers at the bottom of the pyramid might encourage other companies to do the

same, even if they were not in the same sector or location. Consequently, challenge funds should

look for unexpected changes, and revisit their definitions in the light of experience.

5.2 Assessing systemic change

5.2.1 Understand the market system

A strong assessment of systemic change will require

some knowledge of the market, both before and after

the intervention. This may require additional information

gathering by the challenge fund, potentially as part of the

grant award process. The challenge fund will also need to

clearly define what type of change is expected from each

project, and the expected trajectory of change. This can

best be done by including it in the results chain.

For example, ECF includes systemic change on the side of

the results chains. The example on the right shows the

expected systemic change from project that sources

abaca (a fibre derived from bananas) from smallholder

farmers. It shows how ECF expects copying by farmers

interested in supplying abaca, and copying by companies

interested in producing abaca based goods. To see how

this links to the full results chain, see the example in

Section 1.1.

5.2.2 Tips for assessing systemic change

Plan for long-term monitoring. The time needed to observe systemic change varies depending

on the type. For example, local copying by farmers or companies of newly introduced practices

can be observed relatively quickly, perhaps after a year or two. However, new businesses are

likely to take two to three years to enter the sector, as they must observe the success of the

grantee, raise capital, etc. In a shorter space of time it should be possible to see early signs of

change. For example, three years may be too soon to assess actual changes in the legal or

regulatory environment. However, early signs of change might include government officials

drafting new legislation, showing increased openness to discussion, or inviting more businesses

to comment on proposed changes.

Consult the business, but do not ask them to measure systemic change. Discuss systemic

change with the grantee, using accessible language. Simple questions like ‘has any other

company shown interest in or contacted you about your business model’, ‘what new

competition do you have’, ‘what companies do you buy from or sell to’, and ‘what regulations

are causing problems for your business?’ will help gather relevant information. Explain why

systemic change is important for the challenge fund, and businesses will generally be happy to

discuss it, or suggest key informants. However, it is not part of their core business, and so

28

grantees are unlikely to conduct data collection or credibly report against systemic change

indicators.

Take an investigative approach. The fund manager must adopt a flexible approach to

measurement. Accepting that systemic change cannot be fully predicted in advance, a certain

amount of detective work is required to establish what has changed in the market system

following the intervention.

Interview different stakeholders. Speak with multiple informants to triangulate information and

better understand the market system. The choice of interviewees will depend on the systemic

change expected, and this should be outlined in the results measurement plan. For example:

o Business Environment Reform: Interview partners from local and national government,

business associations and other businesses.

o Copying by businesses: Meet with other businesses, business associations, and

potentially financial institutions.

o Copying by farmers: Interview the farmers concerned, local authorities, and CSOs or

NGOs that support farmers in that area.

Use qualitative information. Numbers will not show the full story. For example, the fact that

three companies set up similar businesses to a grantee does not show that there has been a

change in the market. Qualitative information will help demonstrate this change, for example by

exploring why competitors set up businesses, whether they found it easy to obtain loans, and

whether they were influenced by the example of the grantee.

Consider attribution. Systemic change is always due to multiple factors, and it is necessary to

show that the challenge fund contributed to any observed change. The below table suggests

questions which might be useful in assessing attribution.27

Type of systemic change

Questions to help assess attribution

Companies copying grantee business model

Does the business model of other companies have similar features to the grantee business model? Is it likely that other companies knew about the grantee’s project?

How did the grantee (or fund manager)’s promotion or other activities influence other companies?

Changes to the business environment

Were changes likely to have been caused, at least partly, by grantee activities? For example, if the government improved regulations in a grantee’s industry, what did the grantee do that might have encouraged the government to change the regulations?

What were the reasons for observed changes? Ask the grantee and other respondents why they think specific changes have happened.

Farmers copying behaviour changes of beneficiaries

Are other farmers’ behaviours are very similar to the beneficiaries’ behaviours?

Were farmers likely to have seen or known about the beneficiaries’ behaviour changes? How?

`27

Adapted from AECF M&E manual

29

6 Tracking Programme Costs The DCED Standard requires development programmes to measure costs, as well as results.

Consequently, challenge funds should track the size of each grant, as well as costs associated with

allocating and managing the funding.

7 Reporting Results The DCED Standard requires development programmes to report their results, internally and

preferably externally. This is particularly important if the challenge fund aims to influence

governments and the wider development community.

7.1 Acknowledge contributions Challenge funds typically require matching funding for every project. They should report the ration

of challenge funding to matching finance (often known as ‘leverage’).

This raises interesting questions around attribution. Challenge funds are not the only funder of the

projects, so they cannot claim full ‘attribution’ for the results. However, they may believe that their

funding played a catalytic role; if they had not taken some of the risk, the project would never have

happened, or would not have happened at the same scale or timeframe. (This is the claim of

‘additionality’, discussed above.)

In practice, challenge funds should be careful in their reporting to give a transparent and honest

account of their role. They should clearly state that they contribute to results, rather than implying

that they were the only funder. They should also acknowledge the contributions of other

development programmes, where relevant, and explain why they believe that their funding is

additional to existing sources of finance.

7.2 Responsible reporting Results measurement information may be commercially sensitive. In particular, companies may not

wish to share business performance indicators, such as profit and turnover. Consequently,

information should not be shared externally without permission. Key indicators such as profit and

turnover may be aggregated to give an overview of the portfolio without revealing information

about any one company.

30

8 Managing the System for Results Measurement

8.1 Human and financial resources “The biggest change was realising that results measurement is not the role of an M&E advisor, it’s

the role of the country manager.” – Amanda Jupp, Enterprise Challenge Fund

Challenge funds typically have limited resources for results measurement and management.

However, saving money on results measurement staff and activities may be a false economy if the

challenge fund cannot learn from experience or demonstrate its impact.28

Consequently, it is recommended that the fund manager develops in-house results measurement

capacity, including full-time, trained staff. Investing in capacity building saves money in the long run,

as it reduces dependence on external consultants. The results measurement team should include:

One staff member with primary responsibility for each grantee, who can conduct site visits,

coordinate results measurement activities and outsource as necessary. Current experience

indicates that one full time staff member is able to manage between five and ten grantees. Their

role will not just be results measurement, but also include other oversight of the grantee.

Local data collection officers, who speak the local language (where possible) and can gather data

cost-effectively. This role could be contracted out to a local research firm, if managed carefully

by internal challenge fund staff.

A central results measurement team with the technical knowledge to develop and provide

guidance, manage the results measurement system, quality check reports, and provide training.

However, the team can be small, and support project managers rather than conduct the results

measurement themselves. This ensures that the project manager can effectively use results

measurement information to oversee the grantee.

8.2 Using results in challenge funds The fund manager does not have management responsibilities over the business, and so results

management information will not directly feed back into running the business. To be compliant with

the DCED Standard, the fund manager must show that results are used to learn and improve the

programme. The below list suggests some ways in which this could happen:

Disbursements. If grants are provided over time, the fund manager can withhold disbursements

if activities have not been conducted, outputs not produced, or outcomes not achieved

according to the contract.

28

Callan, M, and Davies, R., 2013, When business meets aid: analysing public-private partnerships for international development, Development Policy Centre

ECF’s experience with mentors

The Enterprise Challenge Fund hired a ‘mentor’; a consultant with experience in the DCED

Standard, who could work long-term with the challenge fund to build and maintain a DCED-

complaint results measurement system. This helped quickly establish a stable results

measurement system – which is vital to maintain the buy-in of the business and challenge fund

staff.

31

Determining future funding. In some challenge funds, the fund manager can provide additional

funding to high-performing grantees. This should be based on an assessment of results to date,

the additionality of further funding, and the risks of crowding out unsubsidised businesses.

Assist businesses. Results measurement should be useful for the business. The fund manager

should stress these synergies and work with businesses to help improve their work. However,

the fund manager typically has limited power; they can provide suggestions or advice, but not

require the business to take action, unless it relates to a contractual obligation.

Improving fund management. Information on results enables improvements to be made to fund

management. For example, if the fund manager finds that businesses with a diverse network of

linkages tend to perform better than those without, then it might provide additional assistance

in building these crucial linkages. Similarly, information from early rounds can be analysed to

identify factors in the business and business models that contribute to success. This analysis can

inform subsequent rounds of funding, such as what businesses are eligible, how marketing is

done, and the criteria for approval.

Learning. Learning can be shared with policy makers, financial institutions and the wider

development and business community to show what works in enterprise development, how

market systems can change and how barriers can be overcome. For example, the results

measurement system in the Business Innovation Fund links to the Practitioner Hub for Inclusive

Business, which aims to provide a space for practitioners to connect, share experiences and gain

new insights to help their inclusive business ventures grow.29

Advocacy. Challenge funds are in a good position to advocate for improvement to the market

system to governments, donors, and other market actors.30 By bringing together donors and the

business community, the fund manager builds a rich knowledge of the constraints facing

businesses in the region, and can use the examples of grantees to build the advocacy case.

It is essential to institutionalise the results measurement system, and ensure that the information

collected is used in the above manner. Three important ways to do this are:

Develop a results culture. Results measurement will not work without a strong interest in

results, which must be driven from the top. Senior management should encourage a culture of

honest enquiry, where staff can critique the information they receive and report failures as well

as successes. Similarly, project managers should develop an open relationship with their

grantees, accepting and learning from the challenges that they face.

Make roles and responsibilities clear. Make the roles and responsibilities clear for both the

business and the fund manager, so both are aware of what results measurement tasks are

expected of them.

Systematise the use of information. Ensure that information collected during monitoring visits

and other studies is fed back to businesses, so that they can use it in their work. For example,

the fund manager may meet with the business after each field visit to review the results chain,

discuss what they have found at each key step, and whether there is room for improvement.

Within the fund manager, there should be time and space to discuss the results achieved and

adapt plans accordingly.

29

See http://businessinnovationfacility.org/ 30

Chilver, A., van Diermen, P., and Jones, W., 2006, Using Enterprise Challenge Funds to Promote a More Enabling Environment for Business: Challenges and Opportunities

32

Annex A: Useful References Select the blue hyperlinks in the text below to download the relevant resource.

8.3 Overviews of challenge funds Callan, M and Davies, R., 2013, When business meets aid: analysing public-private partnerships for

international development, Development Policy Centre.

Heinrich, M., 2013, Donor Partnerships with Business for Private Sector Development: What can we

learn from experience? Donor Committee for Enterprise Development.

8.4 Specific challenge funds Enterprise Challenge Fund. Their website contains a number of useful files on measuring the results

of challenge funds. In particular, select ‘Publications and Reports’, which contains research and

reviews, including a report on their experience with the DCED Standard.

Innovations Against Poverty. Although IAP is not using the DCED Standard, their website contains

some interesting reports and information on reporting results.

Other Challenge Funds: AECF and FRICH both apply the framework of the DCED Standard, but do not

contain detailed information on results measurement on their website to date (accessed June 2013).

8.5 Information on the DCED Standard The DCED Standard Website contains extensive guidance, case studies, and links to further support.

8.6 Other useful resources Coffey International Development, 2012, M4P and Women’s Economic Empowerment, Phase 2:

Guidelines for Incorporating WEE into M4P Programmes, M4P Hub.

Iris Indicators. The IRIS indicators are a catalogue of performance metrics that leading impact

investors use to measure social, environmental, and financial success, evaluate deals, and grow the

credibility of the impact investing industry. They are discussed in more detail in the Defining

Indicators section above.

Jackson, E., 2013, interrogating the theory of change: evaluating impact investing where it matters

most, Journal of Sustainable Finance & Investment, 3:2, 95-110.

Nethercott, K., Jago-Bassingthwaighte, M., and Jupp, A., 2013, Women’s Economic Empowerment –

Practice and Policy Implications from the Enterprise Challenge Fund, Coffey International

The Practitioner Hub for Inclusive Business has a number of valuable resources for practitioners in

challenge funds, including an impact network and a library on measuring impact and results.

Ruffer, T and Wach, E., 2013, Review of M4P Evaluation Methods and Approaches, ITAD. This paper

reviews evaluation of systemic change, including an example from the Enterprise Challenge Fund.

The World Business Council for Sustainable Development (WBCSD), 2013, Measuring socio-economic

impact: A guide for business. The WBCSD has developed a guide for businesses who wish to measure

and manage their impact on society. It proposes a monitoring system similar to the DCED Standard,

based on a results chain and indicators. More information can be found on their main website.

33

Annex B: Systemic Change Definitions from AECF31 Type of Systemic Change Example Copying by other businesses. Other businesses see the benefits of the grantee’s business model, and so copy the idea.

The challenge fund provides seed finance to support an outgrower scheme, which purchases tomatoes from poor smallholder farmers. This is a financial success, and other companies copy the business model and begin to work with smallholder tomato farmers.

Crowding-in. Other businesses are encouraged into the space created by the grantee. The distinction between this and the previous category is that other businesses don’t copy the business model, but offer supplementary services which are only viable because of the AECF grantee.

The challenge fund provides a grant to a seed supplier to set up shops in rural areas. A financial service provider, not funded by the challenge fund, works with the seed supplier to provide microfinance to farmers who wish to buy the seed.

Copying successful practice. People who are not working with the project copy the behaviours or technologies that the project introduced. While the previous two categories refer to behaviour change in businesses, this refers to behaviour change among farmers and others.

The challenge fund provides a grant to an outgrower scheme, which teaches sustainable farming techniques to participating farmers. Other nearby farmers copy these techniques and thus improve their yield.

Business Regulatory Environment. All projects work within a regulatory environment, principally defined by the government. They must follow laws and regulations, and work with government officials to gain permission to work, export, etc. Many companies seek to improve the regulatory environment, to make it easier for them to do business.

The challenge fund provides a grant to a number of livestock businesses, which imports vaccines. Regulations for importing vaccines are time-consuming and cumbersome to follow, and government officials regularly ask for bribes. The businesses join to advocate to the government, contributing to a change in the regulations.

Factor and other market systems. Changes in factor market systems are changes that the project causes in the main factor market systems of land, labour and capital, but also include ancillary markets such as information.

In an above example, financial services organizations provided financial services to customers of a seed supplier. This is 'crowding in', as described above. However, if those organizations also begin to provide financial services to other people and businesses unrelated to the grantee, this indicates a change in the financial market system, as there is improved access to finance.

Innovation The grantee introduces additional innovations - building on and improving the project / business model. These are innovations which were not in the original business plan, but which the business developed as a result of the AECF-funded project.

The challenge fund provides funding to a pesticide company to develop a new type of organic pesticide for a certain pest. Although the original design did not work, it led to the creation of a new type of pesticide effective against a different pest.

31

Categorisation taken from AECF. All examples are imaginary and developed by the author.