Unitaid end of project evaluation: Creating a private ...€¦ · 24/02/2017 · Unitaid end of project evaluation: Creating a private sector market for quality-assured mRDTs –

Unitaid end of project evaluation: Creating a private sector market for quality-assured mRDTs

June – August 2016

FINAL REPORT

Unitaid end of project evaluation: Creating a private sector market for quality-assured mRDTs – June – August 2016

1

EXECUTIVE SUMMARY

INTRODUCTION

Unitaid made a USD 34 million grant1 to support the creation of private-sector markets for quality-

assured rapid diagnostic tests for malaria (mRDTs) in five countries (Kenya, Madagascar, Nigeria,

Tanzania and Uganda) between 2013 and 2016.

The project sought to address a major gap in how the private sector diagnoses and treats fever. In

each country, private providers accounted for between 31% and 66% of fever treatments.2 In 2010,

Unitaid and other donors had launched the pilot of the Affordable Medicines Facility – malaria

(AMFm)3 which funded the provision of affordable quality-assured, subsidised artemisinin-based

combination therapies (ACTs) through the public, private not-for-profit and private for-profit sectors.

As a result, customers could readily access affordable ACTs through private health outlets such as

clinics, pharmacies and drug shops. However, a variety of obstacles prevented private outlets from

offering diagnostic tests for malaria.

Challenges inhibiting private providers from using mRDTs included ingrained consumer and

provider behaviour, lack of quality assurance, and restrictive policy environments. Unitaid funded

activities to address these challenges in the five target countries and the documentation of evidence

to support markets catalysis in other countries.4 The project’s theory of change assumed that (i)

increasing availability of quality-assured affordable mRDTs, (ii) building consumer awareness, (iii)

training providers, and (iv) ensuring an enabling regulatory and policy environment would lead to an

uptake in the use of mRDTs. This in turn would lead to improved fever treatment decisions.

This evaluation assesses the achievements of the project and draws lessons for the future. Unitaid

commissioned Dalberg Global Development Advisors to conduct the evaluation. The evaluation

combined desk research using documents provided by Unitaid, PSI and MC and other grantees,5

interviews with 49 stakeholders6 and visits to Madagascar and Uganda. The evaluation methodology

is detailed in Annex 4.

OVERVIEW OF FINDINGS

Dalberg perceives that the project had three overarching objectives:

1. Prove the concept of mRDT provision in the private sector and determining which

approaches and interventions are effective

2. Support the development of sustainable private-sector mRDT markets in target countries

3. Build and disseminate knowledge and tools to inform the development of other mRDT

markets

1 USD 34 million was the original grant budget. This was revised in to USD 21 million in late 2015 and early

2016. Actual expenditure was USD 20 million as of June 2016. 2 DHS Malaria Indicators Survey data, cited in Unitaid end-of-project results presentation, June 2016.

3 The pilot was implemented in seven countries: Ghana, Kenya, Madagascar, Niger, Nigeria, Tanzania and

Uganda. The AMFm was launched in 2010 and ran through to 2013. Source: http://www.unitaid.eu/en/amfm 4 In time, this may include mRDT markets in other endemic countries (identified by Unitaid as potential impact

of the project) and markets for rapid diagnostic tests for other diseases (additional potential impact of the project). 5 The principal grantee (PSI) and sub-grantees (MC, FIND, WHO and JHSPH) are all referred to as “the grantees”

throughout this report. 6 Including Unitaid, grantees, other organisations involved in in-country implementation, and external

stakeholders.


2

The project had different levels of success across its objectives. With regards to the first objective,

the project created a large body of insights and learnings that helped build the proof of concept and

learning. The project demonstrated the safety and effectiveness of mRDTs in the private sector, and

showed in which contexts private sector distribution is more challenging to implement (e.g. with

regards to required subsidies) and mitigating steps that might be required. On the second objective,

there was partial success. Providers were trained, mRDTs were imported, and consumers bought

and used mRDTs, although at lower levels than expected. The supply chain has been strengthened in

some countries and there has been permanent policy change in Madagascar. There is high likelihood

that the markets that have been developed to date in Kenya, Tanzania and Uganda will continue at

their current size. It is possible that they might expand further if support from other donors

materialises. In Madagascar, support from the Global Fund might enable the continuation of the

market. The project had little long-term influence in Nigeria. Regarding the third objective, a range

of documents are being prepared to draw out and share insights from the project; these will likely

inform private sector engagements in other countries. The figure below summarises the key

achievements of the project.

Figure 1. Key changes to which the project contributed

SUPPLY

The project increased the availability of affordable quality-assured mRDTs to some degree, but we

estimate that approximately half of the mRDTs procured remained on wholesalers’ or retailers’

shelves as of the end of the project.

Availability: Of the 3.1 million quality-assured mRDTs procured by Population Services International

(PSI) and Malaria Consortium (MC), an estimated 1.3 to 1.6 million were used by customers at

3

Direct

impacts

Development and dissemination of a "roadmap" for public-private engagement in malaria case management

Affordable, quality-assured RDTs continue to be made available in the private sector by leveraging the systems, learning and evidence base generated

Increased availability of affordable, quality-controlled RDTs in the private sector markets

1Increased providerand consumer informed demand for RDTs in private sector markets

2Improved quality of malaria case management among private providers

3 4Coordinated project learning on the introduction of RDTs in the private sector

5 6

• Increased awareness among participating providers of importance of diagnosis before treatment in three countries (Kenya, Tanzania, Uganda)

• New expectation among many consumers to receive malaria diagnosis before treatment in three countries (Kenya, Tanzania, Uganda)

• Continued affordability of mRDTs for consumers after subsidy withdrawal in three countries (Kenya, Tanzania, Uganda)

• Involvement of new suppliers in private sector mRDT market in three countries (Kenya, Tanzania, Uganda)

• Job aids and quality assurance tools available for new providers entering market

• Proof of safety of private sector use of diagnostics

• New knowledge codification and management systems introduced at PSI

• Establishment of quality assurance systems for mRDTs in the private sector in all five countries

• Permanent regulatory change facilitating diagnosis in private sector outlets in one country (Madagascar)

• 3,251 private outlets trained to use mRDTs to diagnose fevers

• Availability of mRDTs at a price affordable to consumers in four countries (Kenya, Madagascar, Tanzania, Uganda)

• Distribution of ~3.0 million mRDTs to private sector suppliers

• Sales of ~1.3 to 1.6 million mRDTs to consumers

• Introduction of quality assurance mechanisms for the private sector in all five countries

• Data, tools and systems to allow monitoring of care quality

• Provider understanding of how to use mRDTsand how to manage fever cases

Catalytic impacts

Project outputs

[Direct impact not yet achieved]


3

private outlets during the project. There was significant overstocking. Approximately 1.3 to 1.7

million mRDTs remained in warehouses at the end of the project (predominantly in Madagascar and

Nigeria). Several factors led to this overstocking.7 Despite challenges, by the end of the project,

Kenya, Uganda and particularly Tanzania appear to have fostered markets for mRDTs. Sustainable

markets have not yet been created in Madagascar and Nigeria.

Affordability: It was challenging to find an unsubsidised price that was low enough for consumers

but high enough for retailers. The subsidies differed by country but most led to affordable prices

during the project.8 In Kenya, Tanzania and Uganda the markets have evolved and currently have

unsubsidised prices that consumers are willing to pay and that provide sufficient profits to suppliers.

In Madagascar, affordable pricing might not be sustainable when subsidies are withdrawn.

Quality of mRDT kits: In all countries, grantees worked with the public sector to establish quality

assurance (QA) systems for private-sector mRDTs. All mRDTs provided directly during the project

were quality assured through such processes.9

Quality of mRDT handling and care: There was less provider supervision than originally planned.

Nevertheless, most private outlets targeted demonstrated knowledge of the correct steps for

diagnosing malaria with an mRDT.

DEMAND

The project built demand for quality assured mRDTs among consumers. There is evidence of

growth in consumer demand across most countries. The growth was, however, lower than targeted

in Kenya, Madagascar, and Nigeria. Consumer marketing efforts (behaviour change campaigns and

branding of quality-assured products and providers) seem to have contributed to demand growth.

Price was a key driver of consumer demand. In particular, demand was sensitive to the price of

mRDTs relative to ACTs. Behaviour change campaigns appear to have contributed to sustainable

demand in Kenya, Tanzania and Uganda (more than in Madagascar where price remains a

challenge).

Demand appears to have been built among providers. Providers participated due to two reasons:

profit and appeals to professionalism. In countries where single packs were available, providers

preferred them as they found them easier to use.10 The lack of supply of single packs will likely be an

issue. Standard Diagnostics (SD), the only manufacturer of WHO-approved single-pack mRDTs, does

not intend to supply single packs to African private-sector supply chains.11

POLICY AND REGULATORY CHANGE

7 Initial demand forecasts were based on flawed or incomplete market information in Kenya and Nigeria.

UNITAID’s assessment that there was lack of clarity and incomplete information provided on the procurement and management of ‘bundled RDTs’ led to extensive clarification communication between Unitaid and grantees, which delayed approvals and affected procurement timelines, leading to shorter implementation timeframes. In some countries, delays in regulatory approvals or in gaining the support of interest groups constrained execution, reducing the time available for demand building. 8 With the exception of Nigeria.

9 While there were problems with single packs used in all countries but Tanzania, these were identified and

addressed. 10

In Kenya and Madagascar some single packs were available. In Nigeria and Uganda, only single packs were made available through the project. 11

Interview with Standard Diagnostics.


4

Temporary regulatory waivers were agreed with the governments of Uganda and Kenya.12 A

permanent regulatory change was achieved in Madagascar.13 Although Nigeria saw policy changes

and Tanzania is likely to see policy changes, these were not directly linked to the project but rather

to longer-term engagement between government and global health actors, including MC and PSI’s

Nigerian partner. In all five countries, governments have updated policies to include private sector

mRDTs in QA systems and product specifications. In addition, grantees and their partners engaged in

advocacy, evidence sharing and system strengthening in each of the five countries, which has

informed the policy debate and may lead to further policy change in time.

IMPACT

The project’s impact in terms of its reach during implementation was limited. The targeted number

of fevers diagnosed with mRDTs according to the logframe was not reached. The proportion of

fevers diagnosed with mRDTs and treated appropriately varies by country but in many cases

exceeded the original targets at project outset.14

The percentage of fever cases diagnosed using mRDTs in the private sector is above the target of

30% in Kenya, Tanzania and Uganda, but the absolute number of people reached appears to be

below target in all countries.

According to surveys,15 the percentage of positive malaria cases treated correctly was above

targets in all countries, although Dalberg modelling indicates that absolute targets have not

been reached. The high rates of adequate treatment may not be a result of the project. At the

outset, the percentage of adequate treatment of positive malaria cases was already higher than

targets; progress decreased or levelled off over the lifetime of the project in some countries.16,17

The proportion of negative malaria cases treated correctly was high across all countries at

project outset and increased across all countries.18

FUTURE PROSPECTS

It is probable that, if no additional market development activities take place, the markets in

Kenya, Tanzania and Uganda will remain at their current levels while gains in Nigeria and

Madagascar will reverse. PSI gradually changed its role over time in the Tanzanian and Kenyan

markets by moving from direct distribution to support of other actors. This has increased the

likelihood of market sustainability. In other countries, the role of PSI and MC remained the same

12

In each country, there had been restrictions – either in law or in practice – on certain private sector providers such as pharmacies and drug shops playing a role in diagnosis. 13

The project did not directly target policy change but aimed to generate evidence to support later policy change. 14

Grantees report that Unitaid agreed to consider only proportional targets and not absolute targets, given the difficulties of recording or modelling the total number of fever cases treated, but that this decision was not reflected in Unitaid’s documentation. This report considers absolute as well as proportional targets to assess the magnitude of the impact achieved. 15

Project survey results provide insufficient and inconclusive data on whether treatment decisions improved for both positive and negative malaria cases as a result of the diagnoses. As a result, we are cautious about our ability to draw conclusions from these survey results. 16

Round 1 surveys exceeded first-year targets for under fives in Nigeria, Tanzania and Uganda, and for people five and over in Madagascar, Tanzania and Uganda. 17

Achievement against this indicator fell over the course of the project in Nigeria and Tanzania for children under five, and in Kenya and Tanzania for people five and over. 18

The high rates of adequate treatment were already higher than targets at the outset.


5

over time. The abrupt end of Unitaid funding may have jeopardised market development in the

absence of other donor funding.19

Taking into account potential work by other actors in developing these markets, future prospects

are more positive. In all countries relationships have been built and governments and private

stakeholders have committed to driving change. Moreover, other donors have indicated interest in

taking forward private sector markets in Madagascar, Nigeria and Tanzania.

Other private-sector markets for mRDTs might be catalysed through the roadmap and learnings.

The WHO roadmap is intended to provide recommendations to national malaria programs on how to

build private-sector mRDT markets.20 Moreover, a range of materials were developed or will be

developed to support other actors that engage in diagnostics markets.

OPERATIONS

Grantee selection: Unitaid did not have tools in place to fully assess grantees’ capacity, and hence

understand where project structures (e.g. delegation of authority, risk management) might need to

be adjusted. Unitaid has already established new processes to more critically analyse its grantees’

capacity.

Defining the grant and its overarching objectives: As discussed above, the project had three

implicit, overarching objectives. There was a degree of tension between these implicit objectives,

particularly between the first two objectives. Given this tension and the inherent trade-off, it was

critical that Unitaid and the grantees had a shared and aligned understanding on how to manage this

balance. However, Unitaid and its grantees had different understandings of the weight of each

objective and the balance between them. Despite attempts, these different understandings were

not fully reconciled during project planning and as a result continued to play a role during design,

implementation, and reporting.

Setting national targets and planning country operations: The logframe was not contextualised and

in many cases did not have appropriate targets or a realistic timeframe. Grantees’ project planning

was comprehensive and addressed the full range of relevant issues. However, there were

weaknesses in the evidence gathered by grantees to inform project plans. Grantees developed

strategies to recognise and manage risks that are inherent to market development but these did not

address all relevant risks due to the incomplete market information. As they define project

logframes and targets, both Unitaid and grantees need to ensure that logframes are responsive to

the facts on the ground, and that a good understanding of the market informs planning.

Delegation of authority: As the funder, Unitaid had to define which decisions it would delegate and

which ones it would maintain. Unitaid’s decision to retain authority for approving procurement

reduced the overall project risk ex-ante, and turned out to be an important risk management tool

during the project. However, it did cause bottlenecks in project implementation. Unitaid’s approval

processes were relatively slow and the underlying criteria were sometimes unclear to the grantees.

It is difficult to separate the decisions on the delegation of authority from the grantee selection and

the capabilities and risks of grantees. Going forward, Unitaid will likely want to make an integrated

19

In Madagascar, private sector interventions to support the mRDT market introduced under the Unitaid grant have been rolled into a project funded by the Global Fund. DFID continues to invest in private sector case management in neighbouring counties in Kenya. In Uganda, a six-month transition period is operating as the project closes out. 20

There are delays in finalising the roadmap.


6

decision on grantee selection, project structure, and the possible levels of delegation of authority, to

determine the optimal balance between risk, flexibility and speed.21

Monitoring and reporting: Grantees completed routine monitoring and evaluation, with some

challenges. The extensive use of surveys also posed practical constraints to grantees. Moreover,

inventory data was not systematically gathered on a project-wide basis until June 2015.22 Grantees

submitted detailed semi-annual procurement, financial and programmatic reports although there

were some inaccuracies. Unitaid felt that grantees did not proactively share information on

emerging risks and challenges until problems became apparent. Going forward, Unitaid could further

emphasise the norms and expectations that it has with regards to the proactive sharing of challenges

and issues by its grantees.

CONSIDERATIONS ON THE NO-COST EXTENSION

Unitaid denied grantees’ request to continue operations for six months past the original end date

(the “no-cost extension request”). This decision was based on Unitaid’s assessment of project results

to date.23 However, not extending the timeline reduced the time available to grantees to operate at

full scale, given delays in project set-up. This may have reduced grantees’ ability to achieve some,

but not all, of the project’s goals:

1. Prove the concept of interventions in private-sector mRDT markets: The project generated a

large body of learnings on the challenges and possibilities of introducing mRDTs in the private

sector. Granting the no-cost extension might have generated further learnings on challenges

specific to national scale-up and further enriched the learnings with further details and insights,

but many of general insights on private sector scale-up were likely already captured.

2. Support the development of sustainable private-sector mRDT markets in target countries: The

implementation of pilots was partially on track in most countries and some pilot targets had

been achieved. The scale-up was only partially implemented and most scale-up targets were not

achieved. Despite limited scale-up, some markets grew and appear to be sustainable at local

scale, the no cost extension could have increased the scale further.

3. Build and disseminate knowledge and tools to inform the development of other mRDT

markets. The project resulted in knowledge and tools that could be used by other actors,

including the roadmap. However, the richness of content could have been further enhanced if

the project duration was extended.

VALUE FOR MONEY

Value and impact achieved to date: The project has proven the concept and possibility of incubating

private sector markets for RDTs, resulted in some sustainable markets at sub-national scale, and it

has generated many materials that capture the project learnings and that are starting to be adopted

by other actors. In terms of impact, the project funded the distribution of mRDTs to private-sector

customers who might not otherwise have had access to diagnostics. Given that an estimated 1.3 to

1.6 million mRDTs were distributed, the project has likely averted hundreds of thousands of cases

where non-malarial fevers would have been treated with antimalarials.24 As a result, unnecessary

21

In extremis, Unitaid might decide that certain setups are not possible with some or all possible grantees. 22

PSI and MC had been tracking stock separately since 2014. 23

Unitaid based its decisions on UNITAID’s assessment of (i) the complex project implementation challenges faced and the lack of reliable mitigating actions, (ii) the inclusion of Nigeria, where the project faced elevated challenges, and (iii) unsatisfactory mitigating actions/strategy by the grantee. 24

The majority of fever cases diagnosed with mRDTs were negative for malaria, according to project reporting.


7

use of ACTs was reduced in target areas. This is likely to lead to reduced stock-outs of ACTs and

therefore to improved health outcomes for malaria patients who would otherwise not have had

access to ACTs. The cost of achieving this immediate impact is approximately USD 11 to 14 per mRDT

distributed.

Value and impact in the medium term: The project has also built momentum which will extend the

value generated across the above-mentioned areas. In terms of impact, if recent sales volumes are

maintained in Kenya, Tanzania and Uganda, future sales of mRDTs in the private-sector market could

increase, ensuring longer-term availability of mRDTs to consumers. Over the coming five years,

expenditures per mRDT distributed would fall to below one dollar as the number of sales to

consumers in Kenya, Tanzania and Uganda increases.

Value and impact in the long term: In time, the project may also inform the development of private-

sector mRDT markets in other countries. The research has also generated knowledge and processes

that will improve the effectiveness of grantees, Unitaid, and potentially the sector. Finally, it is likely

that the lessons and hence impact of this study will be used in other health-commodity markets.


8

CONTENTS

EXECUTIVE SUMMARY ............................................................................................................................ 1

ACRONYMS AND ABBREVIATIONS USED ................................................................................................ 9

1. INTRODUCTION ............................................................................................................................. 10

2. FINDINGS ON PROGRAMMES ....................................................................................................... 13

2a. Ingoing situation and need for the project ........................................................................... 13

2b. Supply creation (availability, affordability, quality) .............................................................. 15

2c. Demand creation (consumers and providers) ...................................................................... 21

2d. Policy and regulatory change ................................................................................................ 24

2e. Impact and outcomes of the project .................................................................................... 26

2f. Future prospects for the mRDT market ................................................................................ 31

2g. Learnings ............................................................................................................................... 36

3. FINDINGS ON OPERATIONS ........................................................................................................... 39

4. CONSIDERATIONS ON THE NO-COST EXTENSION ......................................................................... 47

5. VALUE FOR MONEY ....................................................................................................................... 50

6. CONCLUDING THOUGHTS ............................................................................................................. 55

ANNEX 1. THEORY OF CHANGE ............................................................................................................. 56

ANNEX 2. ASSESSMENT OF LOGFRAME ................................................................................................ 57

ANNEX 3. COUNTRY PROFILES .............................................................................................................. 60

3a. Kenya ..................................................................................................................................... 60

3b. Madagascar ........................................................................................................................... 61

3c. Nigeria ................................................................................................................................... 62

3d. Tanzania ................................................................................................................................ 63

3e. Uganda .................................................................................................................................. 64

ANNEX 4. EVALUATION METHODOLOGY .............................................................................................. 67

ANNEX 5. MODEL METHODOLOGIES .................................................................................................... 70

5a. Model of stock flows ............................................................................................................. 70

5b. Model of stock value ............................................................................................................. 71

5c. Calculation of confidence intervals for survey data at goal and purpose level .................... 71

5d. Model of number of malaria cases diagnosed with mRDTs and treated with ACTs ............ 71

ANNEX 6. PERFORMANCE AGAINST LOGFRAME INDICATORS ............................................................. 73

ANNEX 7. ESTIMATIONS OF STOCK FLOWS .......................................................................................... 91


9

ACRONYMS AND ABBREVIATIONS USED

ACT Artemisinin-based combination therapy

ADDOs Accredited drug dispensing outlets

BDH BDH Laboratory Suppliers Uganda

CHAI Clinton Health Access Initiative

DALY Disability-adjusted life year

DFID UK Department for International Development

FIND Foundation for Innovative New Diagnostics

KES Kenyan shillings

MC Malaria Consortium

M&E Monitoring and evaluation

MOU Memorandum of understanding

mRDT Malaria rapid diagnostic test

NDA Ugandan National Drug Authority

NGO Non-governmental organisation

NMCP National Malaria Control Programme

PATH PATH, formerly the Program for Appropriate Technology in Health

PMC Premier Medical Corporation

PSCM Global Fund Private Sector Copayment Mechanism

PPMVs Patent and proprietary medicine vendors

PSI Population Services International

SD Standard Diagnostics

SFH Society for Family Health

SUNMAP Support to the National Malaria Programme

UHMG Uganda Health Marketing Group

USD United States dollar

WHO World Health Organization


10

1. INTRODUCTION

Unitaid made a USD 34 million grant25 to support the creation of private-sector markets for

quality-assured rapid diagnostic tests for malaria (mRDTs) in five African countries between 2013

and 2016. This grant funded a consortium led by Population Services International (PSI) with the

Foundation for Innovative New Diagnostics (FIND), Malaria Consortium (MC), Johns Hopkins School

of Public Health and the World Health Organization (WHO) as sub-grantees. The project countries

were Kenya, Madagascar, Nigeria, Tanzania and Uganda.26

The project sought to address a major gap in how the private sector diagnoses and treats fever. A

substantial number of patients in malaria-endemic countries seek fever treatment in the private

sector.27 However, private outlets do often not adhere to global and local policies and guidelines for

fever case management. 28,29 Many outlets prescribe or provide treatment without prior diagnosis.

Overuse of antimalarials increases the risk of resistance and of unavailability of antimalarials for

patients that need them. 30 Unitaid sought to address this situation by increasing the use of malaria

diagnostics (specifically mRDTs) by private outlets, and by catalysing functioning markets for mRDTs

in each target country. By improving use of diagnostics, the project intended to improve malaria

case management.

The project’s theory of change targeted the main reasons for limited use of mRDTs in the private

sector. The theory of change for the project assumed that by (i) increasing availability of quality-

assured affordable mRDTs, (ii) building consumer awareness, and (iii) training providers, there would

be an uptake in the use of mRDTs, which in turn would lead to improved fever treatment. Moreover,

Unitaid aimed to influence the regulatory and policy environment to ensure that the markets

created were sustainable. The project also included the development of a roadmap that distilled the

learnings from the five countries to help other countries build mRDT markets. Annex 1 presents

Dalberg’s understanding of the theory of change based on the logframe and conversations with

Unitaid and grantees. Annex 2 outlines Dalberg’s evaluation of the logframe.

Grantees engaged in a variety of activities to support market development by improving supply,

demand and the policy environment. PSI and MC used two different models:

In Kenya, Madagascar and Tanzania, PSI procured mRDTs and provided a range of services to

actors along the supply chain. PSI and MC country offices managed their own procurement

processes separately. A different manufacturer supplied each country office. PSI was able to

procure diagnostic kits at more advantageous prices than individual wholesalers by buying in

large volumes.31 Kits were predominantly procured in bulk packs or “hospital packs”. PSI procured

some individually wrapped mRDT kits (each containing the accessories required for one

25

The original grant budget was USD 34 million. This was revised in to USD 21 million in late 2015 and early 2016. Actual expenditure was USD 20 million as of June 2016. 26

All countries selected had participated in the AMFm. 27

From approximately one third in Kenya to two thirds in Nigeria. MIS/DHS data, Cited in end of project results presentation, 2016. 28

World Health Organization, ‘World Malaria Report 2014’. 29

Malaria Consortium, ‘Managing illness through better diagnosis of malaria’. 30

World Health Organization, ‘World Malaria Report 2014’. 31

PS Kenya was able to secure an additional discount by accessing rates negotiated earlier by CHAI.


11

diagnostic test) in Kenya and Madagascar.32 The stock was then sold to various market players.33

PSI and its in-country partners also provided a range of services to build demand, including

training for providers and marketing to consumers.34

In Uganda and Nigeria, MC procured both mRDTs and services from manufacturers. In each

country MC worked with two different consortia of manufacturers, importers and wholesalers.

MC procured mRDT “bundles” of commodities (individually wrapped mRDT kits) and ancillary

services (such as training, waste management and demand creation). In each country,

manufacturers used their in-country importers and distributors to provide these services. MC was

not directly involved in supporting distributors or providing ancillary services. 35,36

In each country, a multi-stakeholder task force oversaw implementation. National project task

forces included PSI or MC, government stakeholders, WHO, professional associations and others.

National malaria control programmes were involved in planning, securing regulatory approvals, and

project execution (at differing degrees). The division of responsibilities varied in each country.

The project was planned in two phases: a pilot phase and a scale-up phase. Project plans

anticipated an 18-month pilot phase in a small number of regions in all countries, followed by a

scale-up to a larger number of regions in four of the five countries.37 Grantees experienced several

delays, including in signing agreements with national authorities, in receiving procurement approvals

from Unitaid, and in the delivery of mRDTs. The first implementation start date was April 2014 (in

Kenya and Madagascar) and the last September 2014 (in Nigeria), compared to original plans for

approximately October 2013.38 Given the delays at the outset of the project, the time available for

implementation was shorter than originally planned. As a result, in April 2015 PSI made a request to

Unitaid to extend the duration of the project by six months (the “no-cost extension request”).39 The

no-cost extension would have allowed a scale-up phase of approximately one year in four countries.

As PSI and MC waited for Unitaid’s decision, teams in Madagascar, Nigeria, Tanzania and Uganda

began to scale up.

Due to limited uptake of mRDTs, the extension was denied and Unitaid ordered a scale-down of

activities in some countries, and restrictions including budget reductions in others. Unitaid

32

In Madagascar, 60,000 mRDTs were procured in single packs and 716,000 in hospital packs. In Kenya 162,500 mRDTs were procured in single packs and 87,500 in hospital packs. 33

In Madagascar to wholesalers, in Tanzania to regional distributors, and in Kenya at first directly to retailers and later to wholesalers. 34

In Kenya, Madagascar and Tanzania, PSI worked with wholesalers and distributors it had worked with for other health commodities. PSI Tanzania further built trust and guaranteed demand by sharing market data and its stock data with suppliers to prove market viability. 35

By design, in Nigeria and Uganda, MC did not assume control or responsibility for building relationships between distribution partners. Instead, the manufacturers (and the distributors the manufacturers contracted) built demand as they chose. This led to a variety of models. For instance, in Uganda, one manufacturer-distributor consortium paid for services such as waste management in advance, while another paid these in arrears. The latter model inhibited demand by adding short-term costs for wholesalers and retailers. 36

MC provided a subsidy for the mRDTs and the services required by paying a sufficiently high price. This price allowed manufacturers to provide each bundle at a price that was affordable for customers and allowed sufficient margins for all actors. 37

Project plans included a scale-up phase in all countries except for Kenya. 38

Project annual and semi-annual reporting 39

The no-cost extension requested permission to make expenditures of USD 2.9 million in the six months after the original project closure date. These expenditures would have been funded from the underspend in previous accounting periods, and would not have added to the total costs faced by Unitaid.


12

concluded that it would be difficult for grantees to achieve the results expected from the no-cost

extension.40 In October 2015 Unitaid decided not to grant the extension. Unitaid also ordered a

significant scale-down of activities in Nigeria and Uganda, and reduced the project budget

accordingly. In Madagascar and Tanzania (and in Kenya, where no request for an extension was

made), the scale-up phase was limited to approximately six months and budgets were also reduced.

Following the denial of the no-cost extension, PSI and MC have engaged in various activities to use

up excess stock before they expire, including distribution to parties other than private sector

distributors and outlets.41 In total 27,150 mRDTs were recalled for destruction due to expiry.42

Table 1. High-level summary of project activities

Following project closure, Unitaid engaged Dalberg Global Development Advisors to conduct its

final evaluation of the grant. This evaluation assesses what was achieved and compiles learnings.

The evaluation combined desk research,43 interviews with 49 stakeholders,44 and visits to

Madagascar and Uganda. The evaluation was completed between June and August 2016. The

evaluation methodology is detailed in Annex 4.

40

Project planning initially envisaged a pilot period of 18 months followed by a scaled-up period of 18 months. The first 18 months of the implementation period (which started from the date of deployment of mRDTs procured) had not been completed in all the countries when the no-cost extension was requested. 41

This includes working with new private outlets in Uganda and donating mRDTs to the National Malaria Control Programme in Kenya. 42

13,450 mRDTs expired on the shelves in Nigeria and 13,700 in Uganda. These are due to be destroyed. 43

Using documents provided by Unitaid, PSI and MC and other members of the grantee consortium 44

Including Unitaid, grantees, other partners involved in in-country implementation, and external stakeholders.

1

High-level summary of project delivery

2013 2014 2015 2016

AprMayJun Jul AugSepOctNovDec Jan FebMarAprMayJun Jul AugSepOctNovDec JanFebMarAprMayJun Jul AugSepOctNovDec Jan Feb

Provider demandbuilding, training, supervision

Uganda

Activity

Tanzania

Nigeria

Kenya

Value chain analysis

RDT delivery and testing

Madagascar

Consumer demand building

Nigeria Value chain analysis

Tanzania

Nigeria

Uganda

Nigeria

Uganda

Kenya

No-cost extension request

Unitaid visit to Nigeria and Kenya

Madagascar

No-cost extensiondecision

Tanzania

Market research

Kenya

Madagascar

Madagascar

Uganda

Tanzania

Kenya

Exact date of delivery

NB: Excludes ongoing activities e.g. engagement with regulators

Approximate date of delivery


13

2. FINDINGS ON PROGRAMMES

2A. INGOING SITUATION AND NEED FOR THE PROJECT

In each country, private providers treated a large number of suspected cases of malaria. Each of

the five priority countries had a high malaria burden. Combined, they accounted for over 77 million

suspected malaria cases in 2013 – or over 20% of the total of suspected cases globally.45,46 Private

facilities, including small owner-operated drug shops, larger pharmacies and private clinics,

accounted for between 31% and 66% of fever treatments.47

Private providers in the five countries had access to subsidised ACTs but faced obstacles in offering

diagnostic tests for malaria. In 2010, Unitaid and other donors launched the AMFm pilot project.

The pilot was implemented in seven countries: Ghana, Kenya, Madagascar, Niger, Nigeria, Tanzania

and Uganda.48 The AMFm funded the provision of affordable, quality-assured, subsidised ACTs

through the public, private not-for-profit and private for-profit sectors. As a result, affordable ACTs

were readily available in the private sector. WHO and governments recommend that suspected

malaria cases should be diagnosed before treatment.49 This recommendation aims to ensure the

availability of ACTs for those needing them and to reduce the risk of resistance due to overuse of

antimalarials.50 However, the private sector faced obstacles in offering diagnostic tests for malaria.

Obstacles that private providers were facing included ingrained consumer and provider behaviour,

lack of quality assurance (QA), and restrictive policy environments. The Unitaid grant funded

activities to address each of these challenges.

In all countries, consumer preferences inhibited demand for mRDTs. For over a decade

consumers in malarial areas had received the message that presumptive treatment with

antimalarials was the best course of action. This message was based on prior WHO guidelines

that recommended that all fevers in children under five in Africa be presumptively treated with

antimalarials.51,52 ACTs and other antimalarials were often easily available over-the-counter in the

private sector. The AMFm and its successor, the Private Sector Co-payment Mechanism, made

ACTs more affordable. ACTs were often cheaper than diagnostic tests, making presumptive

treatment cheaper. There was also a common assumption that all testing in pharmacies was for

HIV, introducing an element of stigma for consumers.

While mRDTs were available in the public sector, only Nigeria had relevant private sector

distribution of mRDTs. Each country had a public sector supply chain for mRDTs. However, the

45

World Health Organization, ‘World Malaria Report 2014’, Annex 6a. 46

2014 data on suspected malaria cases reported worldwide and in target countries. 47

DHS Malaria Indicators Survey data, cited in Unitaid end-of-project results presentation, June 2016. 48

The AMFm was launched in 2010 and ran through to 2013. Sources : http://www.unitaid.eu/en/amfm, http://www.theglobalfund.org/en/news/2012-11-15_Board_Approves_Integration_of_AMFm_into_Core_Global_Fund_Grant_Processes 49

The governments of all the countries in scope had adopted WHO’s ambition to ensure all malaria cases are tested before treatment. 50

Interviews with the national malaria control programmes of Kenya, Madagascar, Nigeria and Tanzania and the National Drug Authority of Uganda. 51

Unitaid, ‘Malaria diagnostics technology and market landscape’, 2016. 52

D'Acremont V1, Lengeler C, Mshinda H, Mtasiwa D, Tanner M, Genton B. 'Time to move from presumptive malaria treatment to laboratory-confirmed diagnosis and treatment in African children with fever.' PLoS Med. 2009 Jan 6: 6 (1).

http://www.unitaid.eu/en/amfm

http://www.theglobalfund.org/en/news/2012-11-15_Board_Approves_Integration_of_AMFm_into_Core_Global_Fund_Grant_Processes/

http://www.theglobalfund.org/en/news/2012-11-15_Board_Approves_Integration_of_AMFm_into_Core_Global_Fund_Grant_Processes/


14

suppliers and distributors did not supply private providers due to a vicious cycle of low demand

and high prices.

o In four of the five countries, there were limited markets for mRDTs. In Kenya, Madagascar

and Tanzania, there was no significant market servicing private health facilities and

pharmacies with mRDTs. In Uganda, there was a private sector market in private health

facilities and pharmacies, but there was no significant market among smaller drug shops.53,54

Providers who were not used to working with mRDTs did not have experience managing the

services required to operate diagnostics safely, in particular medical waste disposal. They

therefore required training on how to administer and manage diagnostics, including

accessories such as gloves, lancets and buffer. Providers were also reluctant to risk their

business by turning down customers’ requests for treatment in favour of a diagnostic

product customers were not demanding.

o In Nigeria, a private sector market for mRDTs already existed – including a large illicit

market. Initial market research indicated a small private sector market concentrated in

private clinics and laboratories.55,56 During the project, MC and Unitaid discovered a larger

market than anticipated. This larger market-segment included non-quality-assured mRDTs

and illicit supplies leaked from the public sector to the private market. Competitive pricing

was therefore challenging, especially for high-quality products.

National authorities did not perform QA for mRDTs that were intended for private outlets. The

lack of QA compounded the difficulties in building demand, as it increased the risk that kits could

be faulty or of low-quality.

Despite governments’ intentions to increase access to diagnostics, regulatory barriers hindered

private outlets from providing mRDTs. In most countries, laws and regulations restricted private

providers from providing diagnostic tests. In Madagascar, while there was no prohibition, there

was no clear legal basis that allowed private outlets to offer diagnostics, which inhibited

providers from doing so.

53

Sustainability is defined here as use of mRDTs at scale that allows commercial viability. In Kenya, Madagascar and Tanzania mRDTs were available in fewer than 10% of all private-sector outlets in a 2011 survey. In Uganda, mRDTs were available in fewer than 10% of all drug shops. In Nigeria, mRDTs were also available in fewer than 10% of all private-sector outlets but subsequent project experience indicated that there was a larger market than previous research found. 54

Poyer et al., Tropical Medicine, ‘Availability and price of malaria rapid diagnostic tests in the public and private health sectors in 2011: Results from ten nationally representative cross-sectional retail surveys.’ 2015 55

In Nigeria, mRDTs were also available in fewer than 10% of all private-sector outlets but subsequent project experience indicated that there was a larger market than previous research found. 56

Poyer et al., Tropical Medicine, ‘Availability and price of malaria rapid diagnostic tests in the public and private health sectors in 2011: Results from ten nationally representative cross-sectional retail surveys.’ 2015


15

2B. SUPPLY CREATION (AVAILABILITY, AFFORDABILITY, QUALITY)

AVAILABILITY

Figure 2. Estimated stock volumes by stage of distribution, end of project (April 2016)

Of the 3.1 million quality-assured mRDTs procured by PSI and MC, an estimated 1.3 to 1.6 million

were used by private sector customers during the project. 4.9 million mRDTs were planned to be

procured according to original project plans, but orders for 1.8 million were cancelled at Unitaid’s

direction.57 According to Dalberg’s estimates, of the 3.1 million mRDTs actually delivered58 an

estimated 3.0 million mRDTs were distributed to wholesalers, distributors or retailers. Of these,

approximately 1.3 to 1.6 million59 were likely sold to consumers during the project period.60 In all

57

These were intended for Madagascar, Nigeria and Uganda. The UNITAID decision to cancel the orders was

due to (a) lack of prior formal approval of the quantities procured, and (b) low consumption of RDTs,

overstocking at country level and risk of RDT expiry and (c) lack of clarity on the market challenges and the

coordination with country level stakeholders.

58 Worth approximately USD 1.9 million.

59 Worth approximately USD 0.8 to 0.9 million.

The project increased the availability of affordable quality-assured mRDTs by approximately

three million mRDTs, but we estimate that approximately half of the mRDTs procured remained

on wholesalers’ or retailers’ shelves as of the end of the project.


16

project countries stock-outs at outlets were rare and targets for availability were generally

exceeded.61

There was significant overstocking; approximately 1.3 to 1.7 million mRDTs remained in

warehouses at the end of the project – predominantly in Madagascar and Nigeria.62 In Nigeria the

majority of the overstock was sold at a discount or distributed through state programmes after the

end of the project. 13,450 mRDTs expired before they were used and were therefore destroyed. In

Madagascar, the overstock does not expire until 2018 and might still be sold.63 In Kenya 117,000

mRDTs64 were donated to the National Malaria Control Programme (NMCP) for use by the Global

Fund in training private providers and for national distribution to private outlets.65 In Uganda, MC

planned for the destruction of up to 200,000 mRDTs. Most stock destruction was avoided due to a

tenfold increase in sales following the rapid expansion of the project from one district to nine in late

2015. 13,700 mRDTs were destroyed in Uganda.

Unitaid cancelled the procurement of 1.8 million mRDTs due to the overstock. Of the 4.9 million

mRDTs originally planned to be procured, Unitaid denied the procurement of 1.8 million mRDTs for

Madagascar, Nigeria, and Uganda. 66

Several factors contributed to flawed demand forecasts which resulted in overstocking:

Oversupply of mRDTs in the Nigerian market. Grantees did not recognise the extent of public

sector leakage, the availability of low-quality substitutes, and the availability of counterfeits

through their initial research in Nigeria. Additionally, in mid-2015, the Global Fund geographically

expanded a project which donated free mRDTs to private facilities. The expansion included the

three states targeted by the Unitaid project. MC lobbied the Global Fund not to include project

states in its distribution plan but was not successful.67 The Society for Family Health Nigeria (SFH),

60

Dalberg has estimated the volume of mRDT sales to consumers based on available sales data, procurement plans, reporting from private outlets, and qualitative assessments provided by PSI and MC. These are outlined in Figure 2 and the methodology is detailed in Annex 5. 61

It appears that there was no significant leakage or illicit sales to suppliers or providers not enrolled in the project. Grantees are confident that leakage was avoided because most outlets enrolled were small-scale and because there was intensive oversight through training and supervision. Our estimates therefore assume that all stock delivered to wholesalers, distributors or retailers was sold or remains with those suppliers. 62

Worth approximately USD 0.9 to 1.0 million, or about half the expenditure on mRDTs delivered. 63

Dalberg estimates that the stock currently with wholesalers and retailers could be sold to consumers in approximately 23 to 47 months following project closure. At the lower end of this estimate, all mRDTs in stock would be sold before expiry; at the upper end approximately half would be. This based on an assumption of 11,000 mRDT sales per month, derived from average monthly sales per outlet in 2016 and the number of provider outlets currently operative. This estimate does not include potential increases in the number of outlets consumer that might result from the ongoing work funded by the PSCM. 64

Worth approximately USD 68,000. 65

mRDTs are still available for use in the private sector market but costs were not recouped. If they had not been donated, the stock would likely been destroyed as they were due to expire in 2016. 66

In Madagascar, permission for the final round of 283,000 mRDTs was denied by Unitaid after PSI had made the order. PSI Madagascar was liable for payment and sold its stock to PSI Angola to avoid making a loss. In Nigeria, Unitaid cancelled the delivery of one million mRDTs planned for December 2015 following its assessment that the Nigerian market was saturated. In Uganda, 500,000 of the 1.2 million mRDTs procured were not delivered following a similar assessment by Unitaid that the market was saturated. Unitaid clarified that the orders for Uganda and Nigeria had not been approved. 67

A market research commissioned by MC that analyses the Nigerian supply chain notes the existence of the Global Fund donation project and its role in supplying mRDTs to patent and proprietary medicine vendors


17

PSI’s Nigerian network member, implemented the Global Fund grant. PSI and SFH are

operationally independent, and SFH’s understanding of the Global Fund project appears not to

have informed MC’s or PSI’s project planning. PSI interviewees say that it was not feasible for SFH

to use its relationship with the government or the Global Fund to affect the decision to expand

the donation programme.

Delays in procurement approvals. Delays in Unitaid’s procurement approvals affected timelines,

leading to shorter implementation time-frames. These delays were particularly pronounced in

Uganda and Nigeria (reaching four to five months compared to PSI countries). In these countries,

MC procured a novel “bundle” of mRDTs and related services, resulting in extensive clarifying

communication between Unitaid and MC.

Delays in regulatory approvals. In Madagascar, there was a delay before the legal basis for

private-sector mRDT testing was established in 2014. In Tanzania, the Ministry of Health needed

eight months to complete the process to sign a memorandum of understanding.

Other legal restrictions. Tanzania did not allow in-pharmacy testing and the project could only be

implemented in officially recognised private sector health facilities (such as clinics and hospitals).

PSI procured mRDTs in Kenya assuming a reach of 100 outlets, but only 40 outlets were officially

registered and eligible.

Demand was lower than expected. See section 2c on demand.

Despite challenges, by the end of the project, Kenya, Uganda and particularly Tanzania appear to

have fostered markets for mRDTs. In each of the three countries, private sector operators are active

in wholesale, distribution and retail. In Kenya, the supply market now consists of three wholesalers,

five distributors and 168 retail outlets operating in the coastal region. Distributors expect further

market growth.68 In Uganda, 1,502 outlets have been trained, and two of the distributors

participating in the project have increased their share of the market for mRDTs,69 and a Ugandan

manufacturer has started to sell to the Ugandan private sector.70 In Tanzania, the supply market was

self-sustaining by 2015. There were additional sources of quality-assured mRDTs besides PSI. As a

result, PSI stopped distribution activities and shifted to a facilitation role in which it supported

others’ commercial relationships. Three wholesalers, ten regional distributors and 868 outlets are

now estimated to be active in the market.71

Sustainable markets in Madagascar and Nigeria have not been fostered to date. In Madagascar,

the project enrolled approximately 450 private outlets and five wholesalers. However, sales volumes

are lower than anticipated due to low demand. Demand and ability to pay are so low that

interviewees report that without a future subsidy, there is no prospect of a market in the next five

(PPMVs). However, MC interviewees report that they were taken by surprise when the project started reaching PPMVs in project areas. 68

Due to PS Kenya’s DFID-funded work in contiguous Mombasa and Kilifi counties, some of the market

infrastructure developed in the region is also attributable to DFID funding. 69

A third distributor involved in the project left the mRDT market. 70

Astel Diagnostics, which manufactures its mRDTs in Lugogo in Kampala, sells approximately five percent of its stock to private-sector suppliers, according to MC. 71

Data on mRDTs sold by these other source during the project duration is not available.


18

years.72 In Nigeria, the landscape of private providers does not appear to have changed significantly,

and fewer outlets than targeted operate in the market (868 compared to 3,000 targeted).

AFFORDABILITY

It was challenging to find an unsubsidised price that was low enough for consumers but high enough to

give sufficient margin to retailers. Private providers require a minimum margin. In some cases, this

margin made mRDTs unaffordable without subsidy. There were challenges on both the supply side and

the demand side.

Oversupply of mRDTs from the public sector, from the black market, and from other countries

resulted in the availability of cheaper alternatives. In Nigeria, recommended retail prices were

set with reference to ACT prices but without regard to the price of the other mRDTs that were

available. The pricing strategy was affected by unanticipated increases in the supply of mRDTs

due to leakages from the public sector and free mRDTs as result of the Global Fund grant. The

greater availability of mRDTs depressed retail prices for quality assured mRDTs and retailers had

to accept smaller than anticipated margins.

Consumers’ willingness to pay for mRDTs depended on their own ability to pay and on the cost

of ACTs. This is detailed in section 2c below on demand.

There was diversity in the level and form of subsidies by country. Most subsidies led to affordable,

prices with the exception of subsidies in Nigeria. In each market, grantees attempted to set prices

at affordable levels. Grantees used various levels and forms of subsidy. In most countries, goals for

affordability were met with the exception of Nigeria.73 In Kenya, subsidy levels were increased to

ensure affordability. In Madagascar the value of subsidies was held constant and mRDTs remained

affordable throughout the project. In Tanzania, prices remained affordable despite the fact that no

direct subsidy was paid and that the average retail price increased once stock was exhausted. In

Uganda, a high level of subsidy ensured affordability. Due to severe price competition, in Nigeria,

mRDTs were not affordable despite levels of subsidy that were comparable to other countries.

Prices in Kenya, Tanzania and Uganda have since reached levels that consumers are willing to pay

and that provide sufficient profits to suppliers (approximately USD 1.00, 0.40 and 1.00

respectively), despite the end of the subsidies. In Madagascar, commercial prices remain

unaffordable to consumers without subsidies.74

72

The Global Fund is likely to fund price subsidies until 2018. 73

For details on subsidies please refer to country profiles in Annex 3. 74

Nigeria did not reach adequate subsidised pricing during project implementation.


19

QUALITY OF mRDT KITS

All mRDTs provided were successfully quality assured according to processes devised for the

project. Pre-shipment and post-shipment testing was completed on time in all countries.75,76

In all countries, grantees worked with the public sector to establish QA systems for private sector

mRDTs. Project activities improved QA by: (i) increasing the public sector’s capabilities to oversee the

private-sector market, (ii) improving providers' ability to handle and quality assure their stock, and (iii)

building consumers' demand for high-quality products. Grantees engaged national authorities to

develop national private-sector QA systems or to incorporate private sector mRDTs into public sector

assurance systems.

FIND also developed tools to help private providers conduct QA of their mRDT stock.77 These tools

included a new QA technology (positive control wells), a troubleshooting guide describing problems

often encountered with RDTs and solutions, and visual guides to promote appropriate use of mRDTs

by providers.

Problems with single-pack kits were identified and addressed. The buffer solution needed for the

diagnostic tended to evaporate in single-pack mRDT kits, possibly due to the packaging used. 78 PSI,

MC, FIND, WHO and the national authorities in each country collaborated on quality testing to identify

the source of the problem and to understand how many kits were affected. Once affected batches

were identified, the manufacturers issued additional accessory packs with replacement buffer.79 At the

global level, FIND and WHO developed guides for providers on how to identify and handle faulty

packs.80 Based on this experience, WHO changed its guidelines in December 2014 to recommend that

single packs should not be used until they are able to preserve buffer in equatorial conditions.81,82

75

Indicator O1.8. 76

There were obstacles such as procurement delays and challenges in receiving stocks and shipping them to test sites in Nigeria. 77

Such as clinics, hospitals, pharmacies, etc. 78

Produced by all three of the manufacturers involved. 79

Grantees requested that outlets report the number of faulty kits they encountered. Country task forces also led information campaigns so that participating outlets understood the issue. 80

WHO vetted and approved guides developed by manufacturers to help providers assess whether kits contained sufficient buffer. FIND developed a troubleshooting guide for providers to help them resolve problems on-site. 81

As a result, in the remaining procurement rounds (by PS Kenya and PSI Madagascar) bulk packs (also known as “hospital packs”) were procured instead of the planned single packs. In Nigeria and Uganda, where additional procurement rounds were still planned, there was no shift to hospital packs. 82

As of March 2016, WHO has prequalified two single-pack mRDTs manufactured by SD.


20

QUALITY OF mRDT HANDLING AND CARE

Potential risks to public health, which would undermine the longer-term sustainability of market

change, were mitigated.

MC required that that implementing consortia provide waste management services at no extra

cost to participating outlets. Health ministries and professional associations identified that private

outlets were likely to find waste management a challenge. Regular waste disposal services were

included in the price paid by providers. By contrast, in PSI’s countries, providers handled waste

management themselves.

There was less provider supervision than originally planned. Nevertheless, most private outlets

targeted demonstrated knowledge of the correct steps for diagnosing malaria with an mRDT.83 In

all countries but Madagascar, support and supervision targets were missed in at least one year.84,85

Despite lower levels of supervision, programmatic reporting indicates that providers were

knowledgeable about the correct process for diagnosing malaria with mRDTs. As a result, a large

proportion of patients diagnosed as negative for malaria received appropriate treatment.86

The Kenyan project team identified that limited availability of ACTs on the market was a key

constraint for the appropriate treatment of malaria cases. Availability was particularly low in the

period between the release of donor funding for ACTs and the distribution of stock to first-line

buyers. PS Kenya therefore amended project plans to directly procure 30,000 ACTs.87

83

PSI countries used technology to monitor standards of care and target support. Tablet-enabled software was used during the supervision visits at outlets. The software allowed immediate and accurate tracking of outlets. It also enabled trainers and supervisors to target behaviour change communications and supervision towards outlets with the highest patient volumes and/or lowest quality scores. This targeted support may explain why increases in quality of care were achieved despite the fact not all outlets were supervised regularly. There were rapid increases in quality of care as measured by PSI’s scorecard in Madagascar and Tanzania (though not in Kenya, where quality of care was higher to begin with). 84

Indicator O3.2, as presented in Annex 6. 85

In Kenya, Madagascar and Tanzania, provider monitoring systems were introduced by PSI. These systems promoted quality assurance and continuous improvement, and enabled PSI to focus their attention on providers in greatest need of help. 86

This was measured as indicator O3.1, as presented in Annex 6. The proportion of clients testing negative for malaria managed according to the recommended treatment algorithm exceeded targets in every year, and increased between 2014 and 2016 in every country but Madagascar (where it declined from 99% to 93%). 87

In Madagascar, the mRDT project was intended to complement a concurrent project on ACT availability also implemented by PSI. PSI’s ACT project was for a private sector co-payment mechanism (PSCM), funded by the Global Fund. PSI teams collaborated on messaging for both training and consumer marketing purposes.


21

2C. DEMAND CREATION (CONSUMERS AND PROVIDERS)

CONSUMERS (INCLUDING CAREGIVERS)

There is qualitative and quantitative evidence of consumer demand growth in most countries.

According to interviews and sales data, end-user demand grew throughout the project in

Madagascar and Tanzania, and towards the end of the project in Kenya and Uganda.

The growth in demand was lower than expected in Kenya, Madagascar, and Nigeria. As illustrated

in Figure 2, in Kenya, Madagascar and Nigeria there was an over-stock of mRDTs, suggesting that

demand did not reach the expected levels. The picture in Nigeria is less clear, given the flooding of

the mRDT market.88

Gains in demand may have not been evenly spread and have been greater in wealthier areas. In

Kenya, Madagascar, Nigeria and Uganda, interviewees report that increases in demand have been

concentrated among wealthier customers.

Consumer marketing efforts such as behaviour change campaigns and the introduction of mRDT

branding seem to have contributed to demand growth.

In Kenya, Madagascar, Tanzania and Uganda, behaviour change campaigns have built demand.

In Tanzania, demand grew steadily after the introduction of behaviour change communications in

late 2014. In Madagascar, demand also grew at a constant but slower pace once the marketing

campaign was introduced. In Kenya, the introduction of consumer and provider marketing

campaigns in the middle of 2014 coincided with a doubling of monthly sales.89 In Uganda,

monthly sales grew tenfold in the second half of 2015 as compared with the first, when the

project scaled from one district to nine and a consumer campaign was begun for the eight new

districts. In Nigeria, due to project delays, consumer behaviour change campaigns were not

started in earnest and did not result in significant changes in demand. This correlation echoes

interviewee’s opinions that marketing changed consumers’ perceptions of malaria testing and

that consumers have begun to demand testing.

Consumer education seems to have increased demand for quality-assured mRDTs. Logos and

branding to help consumers identify quality-assured mRDTs and participating outlets were rolled

out in all countries but Nigeria.90,91 Interviewees indicated these logos have increased demand for

quality-assured products and for outlets carrying the logo in general. In Uganda, customers have

begun to associate the local green logo with high-quality care, further incentivising provider

88

Given the existing market of lower-cost non-quality assured substitute mRDTs in Nigeria, increases in demand for mRDTs does not necessarily translate to demand for high-quality mRDTs. 89

Comparison of average monthly sales between the first and second halves of the year 2014. Source: Annual narrative report. 90

A consumer-facing logo for quality-assured mRDTs was developed and implemented in Madagascar, Tanzania and Uganda in 2014, and in Kenya in 2015. 91

Logos are registered with the respective ministry of health in each country.

The project built some demand among consumers and providers for quality mRDTs. Demand

remains very price/margin-sensitive.


22

participation. A distributor said of the impact of the logo: “The logo is helpful because before

there were lots of low-quality diagnostics. Now, customers want the product with the logo.”

Price was a key driver of consumer demand. In most countries adequate pricing was achieved with

subsidies. For example, in Kenya, consumer sales tripled in 2015 after the introduction of a subsidy

that brought average retail prices down to the recommended level of KES 60.92 In Uganda, a subsidy

of 82% of the target retail price kept prices below target. In Nigeria, where there is greater

availability of alternative products, the subsidy of 62% of target price was not able to increase

demand.

The price of mRDTs relative to ACTs was crucial. In all countries the recommended retail price for

mRDTs was set with reference to the subsidised price of ACTs (as per the AMFm).93 It is highly likely

that mRDTs need to be cheaper than ACTs for long-term market sustainability. As one wholesaler in

Kenya explained: “ACTs are subsidised but mRDTs are not. So a [low-income] patient has to choose,

do I test or do I treat? So of course they go for treatment. The tests are everywhere. Labels and

logos are everywhere; it’s in your face. The only problem is, does the patient want to test?”

Demand appears to be sustainable in Kenya, Tanzania and Uganda but less so in Madagascar. As

discussed earlier, markets in Kenya, Tanzania and Uganda have reached unsubsidised prices that

consumers are willing to pay and that provide sufficient profits to suppliers (approximately USD

1.00, 0.40 and 1.00 respectively). In Madagascar, pricing may not be affordable once subsidies are

withdrawn.94 Moreover, in Tanzania and Uganda consumer awareness of the importance of

diagnosis, of mRDTs, and of mRDT providers increased by 20% or more.95,96

92

Recommended price levels according to market research. According to PS Kenya mRDT sales data, quarterly sales to wholesalers tripled from 9,595 to 26,280 following the introduction of the subsidy before the third quarter of 2014. 93

In all five markets there is some degree of ACT subsidy, although consumer prices vary somewhat. 94

Nigeria did not reach adequate subsidised prices during project implementation. 95

Based on baseline and end line surveys carried out as part of the project. Indicators O2.1, O2.2 and O2.3 measuring these three issues were below targets in end line reporting. 96

However, there was limited change in the percentage of caregiver asking for mRDTs across all countries (indicator O6.2.). In none of the five countries did this exceed 30%; the targets of 60% have been missed by a wide margin.


23

PROVIDERS

Qualitative insights suggest that demand has been built among providers. Interviewees in all

project countries, with the exception of Nigeria, believe that targeted outlets now have an interest

in providing mRDTs. Interviewees also suggest that demand is channelled towards larger providers.97

Moreover, distributors in Kenya, Tanzania and Uganda98 plan to continue operating in the markets as

they have positive expectations for future demand.

Providers participated due to two reasons: profit and appeals to professionalism.

Profit was a key driver of provider interest. In Kenya, grantees report that many additional

outlets entered the mRDT market when a subsidy was introduced in 2015 to increase their

margin. In Tanzania, outlets’ willingness to stock mRDTs increased when recommended prices

were increased, increasing their profit margin. In Uganda, retailers that originally relied on the

subsidy are now willing to stock unsubsidised mRDTs as they are confident of the volumes they

can sell.

Grantees also persuaded providers to participate by appealing to their professionalism. In all

countries, outreach was done jointly with national malaria control programmes and professional

bodies.99 These bodies stressed the positive health impact that providers could have and

explained that, by participating, providers would be in alignment with national malaria strategies.

These arguments persuaded providers that were unsure of the profitability of mRDTs. In Kenya

and Uganda, where there was extensive marketing and a high-visibility brand for participating

outlets, some consumers also began to associate mRDTs with high-quality care. In Uganda,

enrolled outlets reported increased overall sales due to the perception that they were

trustworthy providers.

Providers preferred single packs, which they found easier to use. Where both single packs and

hospital packs were available (Kenya and Madagascar), providers preferred single packs. The

preference was especially strong among smaller outlets that were not used to managing accessories.

In Nigeria and Uganda, where only single packs were available from the Unitaid project, providers

prefer single packs to bulk packs and appear to have reduced their purchases of mRDTs now that

manufacturers are only supplying bulk packs.100

Despite engagement, some providers prefer microscopy. In Nigeria, many providers maintained a

strong preference for microscopy. This preference was also reported among some providers in

Uganda and Kenya. In some countries, however – notably Madagascar – providers’ initial preference

for microscopy shifted as a result of education and training.

97

Larger, more expensive clinics offer mRDTs but smaller owner-operated outlets are less likely to do so. 98

Interviewees we not able to quantify the demand increase. Data on private providers’ awareness and trust of mRDTs (indicator O2.4) was not reported in programmatic reporting. 99

Professional bodies that represented the professions that were targeted to offer mRDTs (e.g., pharmacists). 100

As of August 2016, WHO recommendations prequalify two single-pack mRDTs manufactured by SD. However, SD has indicated in an interview that it will not supply single-pack mRDTs in African markets in future as they are uneconomical without the Unitaid-funded subsidy.


24

2D. POLICY AND REGULATORY CHANGE

Each country restricted certain private providers such as pharmacies and drug shops from offering

diagnosis before the project began. Governments imposed legal or practical restrictions due to

several concerns. These include potential public health risks of inadequate waste management or

incorrect diagnoses, pressure from stakeholders (particularly lab technicians), and leakages of

mRDTs from the public sector to the private sector resulting in public sector stock-outs.

The project did not directly target policy change but aimed to generate evidence to support policy

change at a later stage. The project recognised that policy change was important for sustainability.

There were no explicit advocacy efforts to permanently change policies. Instead, the project

intended to demonstrate that the private sector could safely use mRDTs. Grantees expected that

this evidence could be used to justify policy changes. WHO was originally planning to support

governments adapt policies and regulations. However, delays in developing the roadmap reduced its

ability to engage government audiences.101

In four countries, policy waivers were obtained. Temporary waivers were granted for mRDTs in

Uganda and geographically-specific waivers were given to project areas in Kenya. In Madagascar and

Nigeria, permanent changes were achieved. In Tanzania, the project intended to obtain the waivers

needed to operate in a category of small retail outlets approved to provide some essential

medicines, known as accredited drug dispending outlets (ADDOs). However, this policy did not

change as anticipated. The project therefore operated in private-sector clinics where diagnosis was

already authorised (but not at pharmacies or similar outlets).

The project contributed to permanent regulatory changes in Madagascar. In Madagascar, there

was no legal basis for private outlets to conduct diagnostic testing. A new regulatory text was

developed to allow private providers to offer diagnostic tests, in particular pharmacies and small

owner-operated drug shops. A single national quality specification for mRDTs was also established

for the public and private sectors.102 Interviewees from the Madagascar public sector and

professional associations cite the involvement of PSI as a catalyst for both changes. PSI’s private

sector experience informed the government’s approach to engaging the sector and its engagement

of a broad set of stakeholders ensured buy-in.

The project might contribute to changes in Kenya and Uganda. Government authorities from Kenya

and Uganda have made commitments (at the end-of-project meeting and since) which suggest that

they aim to extend the temporary waivers. These regulatory changes are not yet guaranteed.

101

Further detailed in section 3. 102

The Madagascar project task force, led by the national malaria control programme, the Direction de Lutte Contre le Paludisme, and the Ministry of Public Health led stakeholders in developing this text. These stakeholders also clarified the product standards that applied to the private sector. Thus, they established a single national specification for the public and private sector mRDTs which was based on WHO standards, and they extended public sector provider quality assurance to provider sector providers.

Temporary regulatory waivers needed for pilot execution were agreed with governments in

Uganda and Kenya, and a permanent change in regulation was achieved in Madagascar. There

may be future permanent changes in Kenya and Uganda. In all of the five countries,

governments have updated policies to include private sector mRDTs in their quality assurance

systems and product specifications.


25

In Kenya, the National Malaria Control Programme (NMCP) is making the case for a longer-term

change. The NMCP is using evidence generated from the Unitaid project to overcome the

objections of professional bodies that are opposed to the change.

In Uganda, the authorisation for pharmacies to diagnose was granted on a temporary basis. The

National Drug Authority has indicated that the findings from this project have increased the

likelihood that it would grant a permanent authorisation.103

Although Nigeria saw policy changes, and Tanzania is likely to do so, these were not directly linked

to the project but rather to longer-term engagement with government.

In Nigeria, owner-operated drug shops104 were authorised to perform diagnoses in February

2015. Nigeria’s National Malaria Eradication Programme reports that this authorisation was

already a priority of the government. MC reports that the waiver was the result of concerted

advocacy by the sector, of which the Unitaid project was one component.

In Tanzania, the National Malaria Control Programme is working to secure waivers for ADDOs. A

temporary waiver for ADDOs was granted for the period July to December 2016 and might form

the basis for a permanent extension. However, this waiver was granted for a different market

development project implemented by CHAI and PSI in eight Tanzanian districts.105 The project will

test the applicability of QA systems (such as that used in the Unitaid project) to ADDOs to ensure

quality of diagnosis and care.

In all five countries, governments have updated policies to include private sector mRDTs in QA

systems and product specifications. All five countries have adopted policies or strategies for QA of

private-sector mRDTs.106 Moreover, all countries have public sector systems in place that can

conduct QA for private providers of mRDTs.107

The project also contributed to a broader shaping of the policy environment across all five

countries. Grantees and their in-country partners engaged in advocacy, gathering and dissemination

of relevant evidence and system strengthening. They report that this has shaped debate in each

country’s health sector and may inform subsequent policy change.

103

Before a permanent authorisation is possible, legislation has to establish the right of the National Drug Authority to make this decision. The legislation is expected in autumn 2016 and policy change following that. 104

Known as patent and proprietary medicine vendors (PPMVs). 105

CHAI is leading implementation in seven districts and PSI in one. This project is based on an earlier CHAI pilot that investigated ways of helping the government to meet its strategic targets of increasing malaria testing capacity by 80%. 106

Such as product specifications and quality assurance guidelines. 107

The goals for each of the three indicators that relate to national quality assurance systems have been achieved (indicators O1.7, O1.8 and O6.1).


26

2E. IMPACT AND OUTCOMES OF THE PROJECT

The grant aimed to have impact by improving the quality of fever case management. Adequate

management included use of mRDTs in diagnosis, and then treatment of both positive and negative

malaria cases according to recommendations. The various concepts comprising adequacy of fever

case management are outlined below in Figure 3.

Figure 3. Indicators on the adequacy of fever case management using mRDTs

In initial versions of the logframe absolute targets were set, these were later dropped. Grantees

report that Unitaid agreed to consider only proportional targets and not absolute targets, given the

difficulties of recording or modelling the total number of fever cases treated. This decision was not

reflected in Unitaid’s documentation or in reporting of progress against logframe indicators.

Nevertheless, to understand the scale of the impact, the remainder of this section assesses the

achievements of targets in absolute as well as proportional terms, recognising the limitations of

these absolute targets.108

UPTAKE OF mRDTs

The percentage of fever cases diagnosed in the private sector exceeded the end-of-project target

(30%) in Kenya, Tanzania and Uganda; however, for all countries, the absolute number of people

reached appears to be below the scale intended.

108

Limitations include (i) the fact that the targets reported in the logframe were not revised to reflect reductions in the number of mRDTs entering the market due to procurement cancellations, (ii) grantees’ understanding that absolute targets would not be applied.

4

Is mRDT used?

Do positive malaria cases receive recommended

treatment?

Do negative malaria cases receive recommended

treatment?

Treatment of malaria cases with ACTs• Indicator G1

Y

N

Y

Treatment of non-malaria cases with appropriate treatment• Indicator O3.1

mRDT use• Indicator P1

Is mRDT result positive for malaria?

mRDT sales• Sales data (PSI and MC sales

trackers)

Information not reported through Unitaid logframe

N

n/a

UPTAKE OF mRDTS

TREATMENTADEQUACY

The project had limited direct impact during implementation as the absolute number of fevers

diagnosed with mRDTs and treated with ACTs was likely less than the number of mRDTs

procured and than initial targets


27

The percentage of mRDT uptake was above target in Kenya, Tanzania and Uganda but not in

Madagascar or Nigeria. According to survey results, the percentage of people seeking fever

treatment at private outlets diagnosed using mRDTs is above the 30% target in Kenya, Tanzania

and Uganda. In Madagascar and Nigeria these targets were missed. Attainment increased year-

on-year for all countries.109,110

The absolute number of sales is significantly lower than the absolute targets set in initial

project planning. The project did not track the absolute number of people seeking fever

treatment at private facilities who were diagnosed with an mRDT. However, a proxy is the

estimated number of mRDTs sold to clients through the project.111,112 No country achieved its

original absolute target for the three years. This can be expected given the compressed scale-up

of approximately six months in most countries. By the end of the project, Madagascar, Tanzania

and Uganda had surpassed their goals for year one, and a significant portion of their goals for

year two. In Tanzania, the total numbers are probably significantly higher than tracked given that

additional demand outside project areas was stimulated.113 By the end of the project, Nigeria and

Kenya had not reached their goals for year one.114

Demand targets were higher than procurement targets. The project aimed for 10.7 million fever

cases to be diagnosed with mRDTs. However, the project only planned to procure 4.9 million mRDTs.

Market catalysis during the three years would have been necessary to achieve the impact goals.

However, in most countries the monitoring systems (both programmatic reporting through the

logframe and sales reporting) did not aim to capture sales of other market actors.115 Grantees were

therefore never in a position to report on the achievement of absolute targets.

109

Attainment increased year-on-year among people five and over in all countries. In all countries but Tanzania attainment also increased year-on-year for children under five. In Tanzania the proportion of children under five that were diagnosed with an mRDT fell between the second and third survey rounds, while still meeting the end-of-project target. 110

Variations in mRDT uptake may reflect variations in prior fever case management procedures and use of diagnostics by provider type and by country. 111

The initial absolute targets for mRDT uptake (combining children under five and people five and over) can be compared to Dalberg’s sales estimates reported in section 2b. 112

As presented in Annex 6. 113

Data does not capture sales outside of the project. 114

However, because there was a pre-existing market in Nigeria, it is highly likely that the total numbers are significantly larger for Nigeria, as numbers do not capture sales outside of the project. 115

Uganda is an exception. MC Uganda conducted a value chain analysis assessing imports of mRDTs to all provider types in the country, and has supported the National Drug Authority’s ability to monitor this data on an ongoing basis.


28

TREATMENT ADEQUACY

The grant also aimed to fund improved quality of treatment for positive and for negative malaria

diagnoses.

Survey results provide insufficient and inconclusive data on whether malaria treatment decisions

improved as a result of the diagnoses. Only a subset of all survey respondents were positive for

malaria and therefore included in this indicator.116 As a result, it is not clear whether any sample so

small (as small as ten survey respondents total in Madagascar) can be sufficiently representative to

accurately estimate the public health impact.117 We are cautious about the ability to draw strong

conclusions from these small samples.118,119

The percentage of positive malaria cases treated correctly was above targets in all countries.

Dalberg modelling based on sales data suggests that the absolute number of fever cases treated

correctly was lower than anticipated in the original absolute targets.120

The percentage of positive fever cases that received effective treatment was broadly above

targets. According to survey results, the percentage of people five and over that tested positive

for malaria and received an effective treatment was above the 60% target in all countries but

Madagascar.121 For children under five, this target was met in Kenya, Madagascar and Uganda,

narrowly missed in Tanzania and missed in Nigeria.

The absolute number of positive fever cases that were treated effectively are likely to be below

the scale intended, based on proxies used by Dalberg. Data on the absolute number of positive

fever cases treated was not reported. However, according to Dalberg’s modelling, the absolute

number of patients that tested positive for malaria and that received effective treatment is likely

to be below the initial targets set in all five countries.122,123,124

116

To determine whether there have been improved treatment decisions, the evaluation considers if people (children under five and people five and over) with fever cases testing positive for malaria receiving effective antimalarial treatment. 117

Although, it is statistically defensible to conclude that targets were met or missed with 95% confidence. 118

Only those diagnosed using mRDTs could be included in the sample. Where caseloads were small or where uptake of mRDTs was low, the ability to reach an adequate target size was constrained. Use of survey data in project reporting suggests that either the donor and grantees failed to anticipate this possibility, or that researchers engaged by grantees failed to sample accordingly. 119

To mitigate low samples, PSI also gathered data on these variables through routine reporting but these were not included in the Unitaid logframe. 120

Details on the methodology for this model and its limitations are outlined in Annex 5 and the results are presented in Annex 6. 121

In Madagascar, survey results are well below targets but sample sizes are too small to make reliable conclusions. 122

The evaluation uses estimates of the number of people (children under five and people five and over) with fevers who test positive for malaria that receive effective antimalarial treatment, as a proxy for the number of improved treatment decisions. 123

Estimates of the number of people (children under five and people five and over) with fevers who test positive for malaria that receive effective antimalarial treatment data are modelled based on (i) the number of mRDT sales modelled above in the supply section, (ii) the proportion of mRDT uses that resulted in positive diagnoses and ACT prescriptions, as indicated in routine sales data (for those sales included in routine data). 124

This estimation has several limitations including: (i) potential for low representativeness of the sample of outlets reporting input data, (ii) inability to correct for seasonal and annual changes in rates of RDT use and ACT prescription, and (iii) inaccuracies in the routine sales data. However, the estimations are very far below


29

The high percentage of adequate treatment, and the falls observed in some countries, may not be

a result of the project. At the project outset, the percentage of adequately treated positive malaria

cases was already higher than targeted. This percentage might have decreased somewhat over the

lifetime of the project in some countries, although this was within the margin of error. Any decrease

may reflect issues outside the direct scope of the project – specifically the availability of ACTs

following the end of the AMFm – which may have reduced outlets’ ability to prescribe the necessary

medication.

In many countries, the percentage of patients receiving adequate treatment was high to begin

with. In Kenya, Nigeria, Tanzania and Uganda, the percentage of ‘people testing positive for

malaria that were prescribed ACTs was already at or above the end-of-project target in the first

survey round.125 By contrast, in Madagascar, targets were not met in any year.126 This suggests

that the blanket end-of-project target of 60% across all countries may not have been appropriate.

Progress reversed somewhat in Madagascar, and levelled off in Kenya and Tanzania.127 In

Madagascar, the percentage of fever cases treated fell slightly between the second and third

surveys, but this fall was within the margin of error.128 In Kenya and Tanzania, this figure

remained approximately the same and any fall was within the margin of error. The rate of

effective treatment dropped in Kenya and Tanzania (where market creation activities were

strongest) but increased in Nigeria where market creation was not achieved. This suggests either

a weak link between project outputs, outcomes, and overall impact, or unreliable survey results.

The percentage of negative malaria cases treated correctly was high at project outset and

increased across all countries. According to surveys, the percentage of negative malaria cases that

were treated correctly in 2014 and 2015 was significantly above the original 30% target in all

countries.129 The end-of-project target was therefore increased to 60% at the grantees’ request; this

target was also exceeded in most countries.130 Nevertheless, stakeholders in all countries report that

a significant proportion of private providers offer malaria treatment without testing. This was

identified as a problem particularly in Madagascar, where consumer demand for tests has been

slower to build.

It is not possible to discern public health impact in terms of mortality and morbidity due to

improved targeting of ACTs due to data gaps. To understand the impact of reduced stock-outs on

mortality would require data on malaria mortality and stock-outs that were not collected or

targets, giving strong reason to believe the actual number of fever cases treated adequately is also likely to be below targets. 125

According to data on indicator G1.2: children over five years old and adults in project areas testing positive for malaria at targeted private outlets that receive an effective antimalarial treatment. 126

In Madagascar, targets were not met in any year for indicator G1.1 (on children under five) or G1.2 (on people five and over). The exception is indicator G1.1 (on children under five in the end line survey) when one child was surveyed who was treated. This gave a result of 100% effective antimalarial treatment. This result is not included in this analysis. 127

Most notably on indicator G1.2 (on effective antimalarial treatment among people five and over). 128

In Kenya and Madagascar, the proportion of cases treated fell among people five and over between the second and third survey rounds, although in Kenya the fall was small and was smaller than the margin of error of the surveys on which this data is based. In Tanzania, the proportion of cases fell among both age groups. 129

Indicator O3.1, as presented in Annex 6. 130

For indicator G1.1, the end-of-project target for children under five was exceeded in Kenya, Madagascar and Nigeria. For indicator G1.2, the end-of-project target for people five and over was exceeded in Kenya, Nigeria, Tanzania and Uganda.


30

reported. To understand the impact of improved mRDT use on resistance would require

epidemiological data that is not available so shortly after project closure.


31

2F. FUTURE PROSPECTS FOR THE MRDT MARKET

FUTURE PROSPECTS FOR MARKETS IN SCOPE

We have analysed future prospects by assigning each country to one of three broad scenarios:

Reversal: The mRDT private sector that has been built is not sustainable; the market that has

been built to date reduces in size.

Continuation: The mRDT private sector market stabilises, with a potential to grow depending on

future changes to the operating environment.

Expansion: The mRDT private sector market continues to grow.

The assessment considers prospects for markets where there are no additional market-shaping

activities. In reality, it is highly likely that additional support from other donors will occur, which

might lead to more positive scenarios. These are discussed later.

It is probable that if no future market-shaping activities take place, the markets in Kenya, Tanzania

and Uganda will remain at their current levels while those in Nigeria and Madagascar will

reverse.131 As seen in previous sections, the projects demonstrated that supply and demand could

be built in most countries. Favourable policy change has occurred or is likely to occur in most

countries. However, success to date at a limited scale cannot be extrapolated to future success at

scale. Future market participants will not enjoy the favourable conditions that attracted pilot

participants: the cheap or subsidised stock, free trainings, product marketing, and (in MC countries)

free services including waste management. Below we provide a summary of our assessment by

country; detailed rationales are in the country profiles.

Kenya – Continuation: In Kenya, three wholesalers and five distributors operate in the market as

a result of the Unitaid and DFID projects. mRDTs are affordable for consumers and profitable for

suppliers. Initial evidence suggests that prices are affordable for consumers even after retailers’

and wholesalers’ costs increased after direct distribution was withdrawn. Consumer and provider

awareness has increased. However, prospects for future nationwide policy change are unclear.

Madagascar – Reversal: In Madagascar, there is less evidence of a vibrant market. Five

wholesalers are operating in the market but they continue to procure stock from PSI. Consumer

and provider awareness remains low outside wealthier segments. Ability to pay is low and there

is consensus among stakeholders that subsidy will be necessary for the next years to avoid a

collapse in demand. The government has made permanent policy changes.

Nigeria – Reversal: In Nigeria, the availability of cheap alternatives to quality-assured mRDTs and

free mRDTs provided by the Global Fund have hampered growth of sustainable demand. Premier

Medical Proportion no longer supplies the market and it has been difficult to attract new

suppliers.

131

The rationale is explained in more detail in the “future prospects” section of the country profiles in Annex 3.

Since the grant finished, in three of the five countries – Kenya, Tanzania and Uganda – there

appears to be an emerging private-sector market for mRDTs. In Nigeria and Madagascar there

is a risk of reversal. Change in other markets might be catalysed through the roadmap and

sharing of learnings.


32

Tanzania – Continuation: In Tanzania, three wholesalers and ten distributors operate in the

market. The supplier market is strong enough that PSI Tanzania was able to stop supplying the

market in 2015. Retail prices rose when the effective subsidy ended but demand was sufficient to

maintain the market. A permanent policy change has not yet been achieved.

Uganda – Continuation: In Uganda, two of the three distributors will stay in the market, and

many outlets report that they intend to continue providing mRDTs. Prices appear not to have

risen after the exhaustion of the subsidised stock. While demand has been generated, the

bundles that outlets prefer are no longer available.132 There is therefore a risk that demand has

been generated for a different product than what the market can provide for. Temporary policy

change was achieved but the prospects of longer-term authorisation are not clear.133

Table 2. Analysis of project achievements and future market prospects by country

Kenya Madagascar Nigeria Tanzania Uganda

Supply

Availability Expansion Expansion Continuation Expansion Continuation

Affordability Expansion Reversal134 Reversal Expansion Expansion

Quality of mRDT

Expansion Expansion Reversal Expansion Expansion

Quality of Handling

Continuation Continuation Continuation Continuation Continuation

Demand Consumers Continuation Reversal Continuation Expansion Expansion

Providers Expansion Reversal Reversal Expansion Continuation

Policy change Expansion Continuation Continuation Continuation Continuation

Expected future scenario

Continuation Reversal Reversal Continuation Continuation

Key

Achievements to date Expected future change

Not achieved Reversal

Partially achieved Continuation

Achieved Expansion

Taking into account potential work by other actors in developing these markets, future prospects

may improve. Relationships have been built and government and private stakeholders have

committed to driving change. At the end-of-project event in March 2016, key stakeholders from all

five countries and from global organisations gathered to share and discuss their experiences as well

132

Some outlets continue to provide mRDTs but do not arrange for sanitary waste disposal. 133

MC reports that the project has contributed to the development and approval of mRDT private sector implementation guidelines at different levels including at the NMCP, and that conversations with senior stakeholders at the Ministry of Health on policy change have advanced. 134

We expect reversal in affordability because of the end of project subsidies. While availability has increased it has not yet reached the point where unsubsidised mRDTs are affordable (this price point seems to be lower in Madagascar than in other countries).


33

as to agree on next steps.135 At the event, national authorities pledged to continue to work on mRDT

policies and strategies, to support marketing and behaviour change communications, and to

coordinate relevant stakeholders.136 By improving the environment through policy change or

continuation of subsidies, other market shapers may contribute to the increased sustainability of all

markets, including those that would otherwise experience reversal.

Moreover, other donors have indicated an interest in taking forward this work. In Madagascar,

Nigeria and Tanzania, other donors may engage in market shaping activities. Thus, the Unitaid

project might lead to the formation of sustainable markets by supporting and informing the work of

others. In Kenya, DFID's parallel project will continue for a year.

In Madagascar some project activities (elements of provider training and consumer campaigns)

will be continued as part of a Global Fund programme.137 In addition, PSI and the Direction de

Lutte Contre le Paludisme have applied for further funding from the Global Fund to cover mRDT

subsidies.138

In Nigeria, the Global Fund is funding SFH to implement a project that donates mRDTs in the

private sector.139 An extension to the project is currently being discussed. DFID will support

ongoing private sector case management through the Support to the National Malaria

Programme initiative (SUNMAP). USAID is investigating opportunities to support states in building

private sector malaria markets. While this would not be an extension of the Unitaid project,

planning has drawn from the lessons learned by MC and PSI.140

In Tanzania, since July 2016 PSI and CHAI have collaborated on a pilot that tests the feasibility of

ADDOs offering mRDTs.141 Evidence generated may inform nationwide policy change that would

allow these outlets to perform diagnoses using mRDTs.

In Kenya, DFID will continue its support for private-sector mRDT markets in neighbouring project

areas for one more year.

135

Attendees included: PSI, MC, WHO, FIND, Johns Hopkins School of Public Health, USAID, the Global Fund, CHAI, DFID, manufacturers, wholesalers, professional bodies for health professions, and the national malaria control programmes of each of the five target countries. 136

The five commitments made by each country are detailed in the end-of-project meeting report. 137

The Global Fund is funding PSI’s project on ACT availability through the PSCM until 2017. 138

PSI will not know whether this additional fund request has been accepted until later in 2016. 139

The project distributes mRDTs for free in Nigeria, and has been discussed in sections above. 140

USAID has also investigated working in Tanzania, although Nigeria has been prioritised recently. 141

These outlets were not included in the Unitaid grant. The CHAI/PSI project is in eight rural districts.


34

PROSPECTS FOR CATALYSING OTHER MARKETS

The WHO roadmap will provide recommendations to national malaria programs on how to build

private sector mRDT markets. There are delays in finalising it.142 The roadmap will include case

studies from each of the five target countries, as well as tools and practical resources. While some of

the intermediate steps were completed on time,143 the roadmap has not yet been finalised due to

delays early in the project and during implementation.144 Much of the content was finalised at the

end-of-project meeting and final dissemination is expected in late 2016.145,146 Some interviewees

questioned if the roadmap would provide a concrete and credible guide to other governments, given

that the ability to reach scale and sustainability has not yet been proven in the five markets.

A range of other materials were or are being developed. Opportunities to develop and share

learning included steering committee meetings, research reports produced and shared during the

project, and the end-of-project workshop which included key external audiences including

government stakeholders. Grantees report that stakeholders in each country (including national

authorities and the private sector) were receptive. In addition, knowledge resources and tools were

developed:

Grantees developed knowledge resources during the project147 and are planning several

additional pieces.148 Since most knowledge resources have not yet been disseminated, it is not

possible for Dalberg to assess how influential they will be. However, an interviewee from USAID

reported benefitting from project learning that was shared less formally. Data and perspectives

shared by PSI and MC have reportedly allowed USAID to make progress in prioritising the

Nigerian mRDT market at a speed that would not otherwise have been possible.

Grantees built tools to support the mRDT market development interventions. These tools

include knowledge and relevant perspectives for market shapers (donors and governments),

142

WHO requested a no-cost extension to Unitaid to allow for completion of the roadmap. Source: WHO report to PSI, ‘Jan-Dec 2015’, 31 March 2016. 143

These included the reviews of existing national policies, were completed according to schedule, as reported against indicator O4.1 and presented in Annex 6. WHO also produced a set of five papers analysing the regulatory environments in target countries to inform advocacy work throughout the project. These were completed on time, meaning targets against indicator O4.1 were met. However, as country-specific reports they are not intended to influence the development of markets in new countries. 144

These delays included recruiting consultants to structure, write and advise on the report, and replacing a consultant that left. In addition, opportunities to engage public sector partners were constrained during the project: there were delays in stakeholder mapping in Nigeria and the political situation in Madagascar made identification of relevant stakeholders challenging. 145

End-of-project report 2013-16. 146

The central objective of the roadmap are recommendations for public-private engagement in malaria case management. These recommendations have not yet submitted to the WHO Malaria Policy Advisory Committee for endorsement. There is therefore no data for indicator O4.2 on submission of these recommendations. 147

Knowledge resources developed include (i) the roadmap, (ii) a chapter for the Springer Encyclopaedia on malaria diagnosis in the private sector, (iii) a set of policy papers with an overview of each country, (iv) two qualitative studies on how mRDTs are used in practice in Madagascar and Uganda by the Johns Hopkins School of Public Health team, which may be adapted for publication in a peer reviewed journal, (v) 9 conference presentations on lessons learned, (vi) a manuscript on quality control of mRDT accessories, and (vii) an article drawing on evidence gathered for the project, ‘Building a healthy and sustainable market for malaria rapid diagnostic tests’. This article was published in the WHO Bulletin. 148

Further work is underway, including (i) case studies laying out lessons learned by PSI during the project, (ii) two further conference presentations, and (iii) four peer-reviewed articles.


35

practical resources for market actors, and new systems to support PSI’s own market development

work in the future.

Grantees have developed tools that market participants can use. These tools include training

materials and job aids, checklists and questionnaires, and FIND’s QA kits for actors throughout

the supply chain.149 Peer organisations including CHAI and USAID see them as useful tools for

their own work. Additionally, PSI has developed a QA manual and toolkit for private-sector. This

was shared at the WHO provider standards meeting in June 2016 and has started to be further

disseminated.

Grantees have improved their knowledge gathering and codification systems. PSI reports that the

new framework it introduced to document and share project learning is effective. Its systematic

process of documenting learning at activity and output stage has now been applied across PSI to

other grant areas. PSI has also developed materials on the lessons it has learned about market

development generally and quality-assured mRDTs specifically. PSI and MC report that insights

derived from this project will inform future work on market creation, and specifically (i) engaging

with markets for the poor, (ii) influencing provider networks, and (iii) assuring quality.

149

The tools were used during the project.


36

2G. LEARNINGS

The project has generated a wealth of learnings on the possibility and practicality of developing

private sector markets for mRDTs. Below we share learnings collected through interviews and

research:

Suppliers need sustainable margins, but these have not yet been achieved in every country. Free

products, leakages from the public sector, and counterfeit substitutes reduced market prices and

suppliers’ margins, particularly in Nigeria and somewhat in Tanzania. During the project subsidies

were needed to maintain competitiveness. With the expiry of subsidies, suppliers will likely see

reductions in their margins.

Training and supervision can increase providers’ trust in mRDTs and their capacity to use them,

but change is not immediate. Providers (clinicians, private sector doctors, pharmacists, etc.) need

to trust the accuracy of mRDT results. Providers have historically relied on microscopy and in some

cases they have had experiences with poor quality mRDTs. As a result, providers do not always trust

mRDT results. The project has demonstrated that trust can be built through training and supervision,

but change is not guaranteed nor immediate.

Behaviour change campaigns help providers and consumers understand that testing is necessary.

WHO recommended presumptive treatment of all fevers as malaria in Africa for over a decade. This

was translated in to public health messaging across malarial countries.150 Diagnosing malaria before

treatment has therefore not been perceived as necessary. The countries where markets have been

built to some extent have seen increased awareness among both providers and consumers due to

behaviour change campaigns. However, achievements have varied widely by country and by

provider type.

Demand is highly price sensitive and pricing needs to take into account the price of ACTs.

Consumers are faced with two options: either paying for an mRDT and potentially for an ACT (if it is

malaria) or another treatment (if it is not malaria) or only paying for ACTs. If the consumer does not

understand the value in testing, the combined price of ACTs and mRDTs is difficult to justify. As

subsidies disappear there is a risk that mRDT prices will rise above ACT prices, and, regardless of

ability to pay, consumers may return to presumptive treatment.

The availability of ACTs influences demand for mRDTs and must be factored into planning. Where

ACTs are largely affordable – as in most countries covered by the AMFm and the PSCM – the price of

mRDTs relative to ACTs is a crucial determinant of demand. In countries where ACTs are less

affordable – as they were at times in Kenya – ACT availability will also affect quality of care.151

Demand likely differs by provider size and by consumer income and education levels, but more

data is needed. Data on these questions was not gathered systematically and should be in the

future. Data on consumer profiles would improve understanding of the grant’s impact in terms of

customer segments reached and the transferability of learnings to new contexts.

150

‘Time to Move from presumptive malaria treatment to laboratory-confirmed diagnosis and treatment in African children with fever’, PLoS Medicine, 2009. 151

PS Kenya observed that price and availability were erratic due to delays between funds being released by donors and ACT supplies becoming available.


37

Public health risks can hinder policy change. Policy makers may worry about changing policy

because inadequate waste management or misdiagnosis could pose public health risks.152 Regulators

are therefore cautious about further expanding the use of mRDTs in private facilities. Both issues

were addressed through training and services during the grant. However, in most countries, less

training and fewer services will be offered following the end of the grant. Provider training will only

continue at scale in Madagascar through Global Fund funding. In Nigeria and Uganda, private

providers had received waste management services at no additional cost but this service has

stopped. Suppliers in Uganda have noted a reduction in the use of commercial medical waste

management services now that providers must pay for it. The public health risks mitigated during

the project may therefore increase following the end of the project, which may reduce the

probability of future policy change.

Pressure from opposed interest groups can also hinder policy change. For instance, in Kenya, lab

technicians perceived mRDTs as a threat to their jobs and lobbied against the use of mRDTs. In

Madagascar, professional organisations representing health professions such as pharmacists

objected to non-pharmacist drug shop owners using mRDTs.

Leakage of mRDTs from the public sector to the private sector can be mitigated by working

together with the public sector. In Tanzania and Nigeria, the existence of a private sector market

had undesired effects on public sector stocks. In both countries there were leakages from public

sector stocks to the private sector. This had negative impacts on both the public and private health

care: the public sector had unexpected stock-outs, while the private sector suffered from higher

volumes flooding the markets and lowering prices. The public sector was an essential partner in

addressing these leakages. For instance, in Tanzania, regulators strengthened monitoring and

tracking, which reduced leakages.

PSI’s gradual step-out of the Tanzanian and Kenyan markets has increased the likelihood of

market sustainability. In Tanzania, PSI changed its role from supplier to facilitator mid-through the

project. This enabled PSI to continue to support the market, bridge relations between manufacturers

and providers, and increase transparency. For instance, PSI Tanzania shared demand forecasts with

suppliers to avert significant price increases. PS Kenya took a step in a similar direction and moved

from directly supplying retailers, to working through wholesalers.

In contrast, project activities ended more abruptly in other countries, which may jeopardise

market development. In Madagascar, there is currently no other market actor that can take on PSI’s

role. PSI will continue to train provider outlets with funding from the Global Fund. Across the other

countries, key market development activities – training of providers and consumer marketing – will

stop, as there is no actor to take on these responsibilities. In Uganda and Nigeria, the bundled

procurement model appears to have ended with the project.153 Now that the bundles have been

exhausted (in Uganda) or expired (in Nigeria) manufacturers do not intend to continue providing the

single packs, and distributors do not plan to provide services such as waste management and

training.

The countries where most was achieved are those where national authorities engaged most. In

Kenya, Madagascar, Uganda and Tanzania, national malaria control programmes led the

engagement of other government arms and championed the project’s aims. For instance, Kenya’s

152

Challenges include taking accurate blood volumes. 153

Procurement from two consortiums of manufacturers and distributors was intended to draw in players who could provide bundles of products and services.


38

NMCP became an ally in engaging the laboratory regulatory agency. In Nigeria the environment was

much more difficult due to the patchwork of relevant authorities at federal and state level, and the

difficulties they and external parties faced in coordinating work.

Three years is likely too short to test, create and scale a market. Market development projects are

typically carried out over five years or more. They require behaviour change at all stages of the

supply chain, from manufacturers and importers to wholesalers and retailers to consumers.

Interviewees at PSI, MC and other members of the grantee consortium say that their ability to

achieve lasting change within three years was limited.154 This belief was echoed by external

stakeholders.

154

Timelines were reduced from the three years planned due to procurement delays at the beginning.


39

3. FINDINGS ON OPERATIONS

The following section follows the program planning and management chain, and discusses key

activities in sequence, namely 1) grantee selection, 2) definition of the grant and its overarching

objectives, 3) setting of national targets and country operations, 4) project implementation and

management, 5) monitoring and reporting.

1. GRANTEE SELECTION

Unitaid did not have tools in place to fully assess grantees’ capacity. Unitaid’s initial assessment of

grantees focused on compliance rather than assessing their implementation capacity. The

verification assessed grantees’ risk controls through a questionnaire but did not independently

assess grantees’ management systems or prior performance in similar grants. As a result, it did not

identify all the potential challenges the grantee consortium could face when working in the high-

uncertainty area of market development. In particular, MC’s limited experience in procuring large

volumes was not reflected in project plans, and was therefore not addressed through a risk-

mitigation plan.

Unitaid has already established new processes to more critically analyse its grantees’ capacity.

Unitaid has introduced a more detailed grantee capacity assessment, which includes an independent

analysis of management structures, and requires enhanced information and risk mitigation where

risk is assessed to be higher (for instance, when subcontracting is used or grantees have no prior

record with Unitaid).

2. DEFINING THE GRANT AND ITS OVERARCHING OBJECTIVES

There was a degree of tension between the project objectives, particularly between proof of

concept and direct market development.155

Running pilots to prove the concept of interventions in private-sector mRDT markets. This

objective implied learning about the effectiveness of interventions to build private sector

mRDT markets, including the safety and effectiveness of the tests in the private sector.

Learning that an mRDT market would not be feasible in a country is almost as valuable as

learning that it works in another country.

Scaling interventions to support the development of sustainable private-sector mRDT

markets in target countries. This objective is to achieve the implementation targets and

scale the market. There is a tension with the first objective: failure and learning from failure

has an explicit value as per the first objective but is undesirable as part of this second

objective. Even within this second objective, there is a trade-off between short-term and

long-term targets. Short-term targets focus on reaching supply and demand targets, while

long-term targets would imply a greater focus on catalysing lasting policy change,

155

As discussed in the executive summary, Dalberg perceives that the project had three overarching objectives: (i) to prove the concept of intervening in private-sector mRDT markets, and determining which approaches and interventions are effective; (ii) to support the development of sustainable private-sector mRDT markets in target countries; (iii) to build and disseminate knowledge and tools to inform the development of other mRDT markets.


40

strengthening the supply chain, and exploring the phase-out of subsidies to ensure the

sustainability of the intervention in the long run.156

Unitaid and its grantees have different understandings of the weight of each objective. While all

interviewees recognise the importance of all three objectives, the relative importance given to each

varies between and within organisations.

Some PSI interviewees report that they understood the primary goal to be proving the safety

and effectiveness of mRDTs in the private sector.

Many PSI and MC interviewees involved in project implementation emphasise the objectives

of the grant as building markets in the target countries.

Some Unitaid interviewees and PSI interviewees describe the ultimate objective as gathering

evidence to influence the policy environment in target countries and elsewhere.

Moreover, the logframe was geared towards the shorter-term implementation of pilot and scale up

interventions, with a focus on direct impact. The indicators linked to market catalysis or operational

research (such as policy change and learning materials developed) measure achievement during the

project rather than longer-term influence.

These different understandings were not fully reconciled during project planning and, as a result,

design, operations, and reporting were not fully aligned. A meeting between all participating

organisations was convened to clarify project goals, but the divergence between documented goals

and participants’ understandings remained. With staff changes, the shared understanding of the

grant started to align more closely with the documented goals and potentially less with the original

ambitions. However, as a result of lack of alignment on the implicit objectives of the grant, Unitaid

and grantees have different perceptions on whether the goals of the grant have been achieved.

There is an opportunity to improve the processes for developing project objectives and conducting

project planning, and to better delineate roles and responsibilities between Unitaid and grantees.

By co-developing the project goals, Unitaid created ownership among the grantees, but also diffused

the responsibility for this deliverable, which might have contributed to the lingering misalignment. It

might have not been understood by all parties who was responsible for articulating the overall

objectives, recognising and resolving tensions between objectives, and translating the objectives

into lower-level indicators.157 There is also an inherent tension in developing project objectives and

targets after the implementing actors have been selected and the project has started. Both the

funder and the grantee have a responsibility for recognising conflicts or tensions between

objectives. Unitaid could have been more precise in the project deliverables and objectives, and

ideally would have pressure tested for possible tensions and trade-offs. The grantees were,

however, responsible for realising, flagging and forcing resolution of tensions and conflicts in the

objectives. As noted above, Unitaid reports that it attempted to resolve misalignments with grantees

early in the project. By contrast, grantees report that they made attempts to discuss and refine the

logframe with Unitaid before and during the project, but that not all of these requests were acted

on.

156

In contrast, subsidies may be a desirable component of the project if the main objective is proof of concept, as it helps to learn about what particular price point is attractive to both consumers and suppliers (even if it might not be sustainable). 157

The logframe did not reflect the importance of the various goals. Several of the implicit goals understood by PSI and/or MC, such as strengthening the supply chain or developing tools and systems, did not have adequate targets and indicators.


41

3. SETTING NATIONAL TARGETS AND PLANNING COUNTRY OPERATIONS

The logframes developed by Unitaid with inputs from the grantees were not contextualised and in

many cases did not have appropriate targets.

Targets did not take into account national contexts and were not always appropriate. For the

majority of indicators, uniform targets were set across all countries and did not take national

contexts into account.158 In many cases, targets were set – and maintained – below the baselines

established in first-round reporting.159,160

Targets did not always have a realistic timeframe. Given the number of actors involved and the

requirement to change a series of ingrained behaviours, the targets may not have been realistic for a

three-year project. Examples include the target of tripling caregiver demand for mRDTs,161 and

various objectives not included in the logframe such as strengthening the supply chain. For

comparison, Madagascar’s national malaria control programme set a five-year timeline for a similar

behaviour change programme amongst public sector health care providers.162

Grantees’ project planning was comprehensive and addressed the full range of relevant issues.

However, there were weaknesses in the evidence gathered by grantees to inform project plans.

The landscape studies PSI and MC commissioned did not address all the aspects needed, meaning

that project plans did not take account of realities on the ground.163 The landscape research was in

many cases based on secondary research, as opposed to up-to-date primary research. For example,

procurement volumes were decided based on epidemiological and demographic data, rather than

on market realities that could affect sales volumes in the short term.164 In future, grantees should

prioritise a thorough market landscape analysis that is based on both primary and secondary

research. A thorough landscape assessment could have helped the teams develop more realistic and

targeted plans. For instance, it might have been unrealistic to expect markets to have been created

in three years when considering the time needed to: sign MOUs with governments, change policies,

train and incentivise private facilities, change consumer behaviour, etc.

158

However, targets referring to a baseline were set for three output indicators: O2.1, O2.2 and O6.2. Targets took the form of improvement against a 2013 baseline. The targeted increase was uniform across all countries. 159

This includes indicators O1.1, O1.2 and O3.1 for all five countries. 160

An exception to this is at the goal level, where end-of-project targets were revised up from 30% to 60% at the request of grantees after early surveys indicated that the 30% target had been exceeded in all countries for people five and over and in most countries for children under five. 161

Indicator O6.2. 162

Interview, Former head of case management department, Madagascar Direction de Lutte Contre le Paludisme. 163

In Kenya, the importance of outlets’ registration status was not reported, and the number of registered facilities was not recorded. As a result, although the initial project plan included 100 outlets, only 40 were actually included in the first phases of the project. In Tanzania, no formative market landscaping exercise was carried out, meaning that the project was operating on assumptions about the shape of the market. Once such a landscape assessment was carried out, the strategy changed considerably. In Uganda, initial research did not identify the lower than expected number of registered outlets in the pilot region. In Nigeria, the pre-existing market for mRDTs was not identified. As a result, project plans mirrored those for other countries where market creation was the issue, rather than tackling low-quality or leaked alternatives, which was the larger issue in Nigeria. In addition, several stakeholders (including from the government and civil society) explained that the three Nigerian states selected for the project, Anambra, Cross Rivers and Ogun, were challenging environments. With low income levels and low density of private providers, these regions may have been less favourable than more populous states such as Lagos, limiting what could be achieved. 164

E.g., the number and size of private sector actors, accessibility and regulatory status of outlets, variations in demand by size or sophistication of provider.


42

As a result, grantees’ risk management strategies and ongoing processes for risk mitigation did not

recognise and manage the full range of risks faced in market development. In their reflections on

the project, grantees identified some problems experienced – in particular, the lack of control over

all market actors and the existence of illicit actors – as inherent to market development work. “In my

experience there are few health commodities that do not pass through this phase: leakage,” said

one country manager. Grantees believe that Unitaid did not recognise and act on the risks flagged in

routine reporting. By contrast, Unitaid felt that grantees were not proactive in flagging these

challenges and making them explicit before they materialised.

As they define project logframes and targets both Unitaid and grantees need to ensure that

logframes are responsive to the facts on the ground. Unitaid needs to ensure that the targets it

expects are realistic and ambitious, while grantees need to ensure that they commit themselves to

achievable goals. Where possible, targets should be derived from baseline data.

Grantees should prioritise a good understanding of the market to enable strong planning. They can

ensure thorough market landscape analysis that is based on both primary and secondary research.

They can also ensure that each target takes into account baselines and a realistic timeframe, even if

some targets might fall outside of the grant timeline.

4. PROJECT MANAGEMENT

Unitaid’s decision to retain authority for approving procurement was an important risk

management tool, but it slowed down project implementation. Unitaid, as any project funder, had

to determine the appropriate delegation of decision-making authority. Unitaid’s decision to maintain

ownership and accountability over procurement decisions was likely correct, given the challenges

with regards to overstocking in the project. However, the inevitable trade-off was that delays caused

by this process delayed implementation of other activities (e.g. demand building). Implementation

start dates were subsequently between six months (in Kenya, Madagascar and Tanzania) and ten

months (in Nigeria) behind original project plans. While grantees conducted some research and

policy engagement in the interim, demand building activities did not start until delivery.

Unitaid’s approval processes were relatively slow, with decision making criteria that were not

always clear to the grantees. Slow approvals affected the project at various points including in

developing a common logframe,165 obtaining procurement approvals,166 and deciding whether the

project would have a no-cost extension.167

Unitaid’s internal processes and high rotation of staff impacted the grant. The main channel for

financial management was ex-post queries to reports submitted. Due to high staff turnover at

Unitaid over the course of the grant, grantees report a loss of institutional memory and

165

There were 12 iterations on the theory of change, and the project was started before all indicators were finalised. As a result, Unitaid and grantees had different perspectives on what had been decided. For instance, grantees reported on performance against percentage targets which they considered conformed with agreements, while Unitaid expected reports on performance against absolute targets. 166

Procurement approvals were delayed for Nigeria and Uganda by approximately four months, as Unitaid sought to fully understand MC’s procurement model. This in turn delayed future activities. 167

At the end of the project, the decision on the no-cost extension took four to five months. Part of this was due to a lack of clarity around requirements on how to file the proposal for no-cost extension. The document PSI considered to be its formal extension request was considered by UNITAID as incomplete: “with a focus on reallocating project funds without providing concrete justifications and value for money.”


43

understanding of problems raised. Grantees also report receiving limited feedback from Unitaid on

the reports submitted.

Unitaid should integrate its decision on delegation and control with the assessment of the

capabilities of all potential grantees. As discussed earlier, Unitaid can conduct a more

comprehensive risk assessment, to further understand, acknowledge and mitigate potential risks.

Using the insights from the due diligence, Unitaid could determine the required amount of oversight

and the delegation needed. On the one hand, more oversight will help minimise risk, but will likely

result in approval delays. It also requires a significant time commitment from Unitaid. On the other

hand, more delegation will minimise delays, require less time commitment from Unitaid, and

enhance grantees ownership. However, it exposes Unitaid to higher risks. It is possible that in some

cases, Unitaid’s perception of the necessary delegation or oversight does not match with grantee’s

preferences or with Unitaid’s own operational model and vision. In such extreme cases, Unitaid

might decide not to work with some grantees or fund some projects (if no suitable grantee can be

found).

Some grantee country teams successfully made course corrections based on new market

information. Some examples of success are due to grantees’ deviations from original project plans in

response to market realities. For example, Tanzania changed its model from purchasing mRDTs to

facilitating the market. Kenya implemented a price subsidy to increase consumer demand and

supplier margins based on its market assessment.

As discussed earlier, there were problems with overstocking in Kenya, Madagascar and Nigeria.

Overstocking was a result of flawed demand forecasts, incomplete scale-up, and nationally specific

challenges. Moreover, PSI’s relationship with SFH did not result in project changes in Nigeria once

SFH started distributing free mRDTs to private outlets as part of the Global Fund grant.168

Interviewees identify a range of reasons for the limited information flows and limited influence on

Global Fund plans, including: (i) SFH’s operational independence from PSI, which reduced PSI’s

insight into SFH’s plans, (ii) siloes between the Global Fund’s public-sector division, with which MC

and PSI had relationships, and its private-sector divisions, and (iii) the sign-off by the Nigerian

government and others actors on the Global Fund plan that included the free distribution of mRDTs,

reducing the Fund’s ability to change its strategy.

Final expenditure was broadly in line with plans; approximately 94% of the revised budget was

spent. Of the final budget of USD 21 million,169 USD 20 million was actually spent.170 Expenditure

against plans varied by budget item; final expenditures ranged from 113% of the budget for output 5

on coordinated project learning, to 30% of the budget for output 4 on the roadmap. This

underspending reflects the delays in project execution and lower than anticipated procurement

volumes. Expenditure against the original budget is summarised below and illustrated in Figure 4.

The largest source of the underspend is output 1, which relates to increasing availability of

mRDTs, including procurement. USD 0.8 million remained unspent.

168

SFH and PSI operate fairly independently and have limited visibility over grants they implement. However, going forward PSI and SFH are working towards more clarity and visibility, as well as strategic cooperation. 169

The original budget was USD 30 million, but this was revised down following the no-cost extension decision and order to scale down activities. 170

Expenditure was reported against each of the project output areas (these output areas are detailed in Annex 1).


44

The largest underspend as a proportion of the target budget was for output 4, which relates to

producing the roadmap. Final expenditures by April 2016 were USD 0.1 million, compared to a

final, revised budget of 0.4 million (which took into account revised project plans and the delays

faced in the work plan for this output). For this output, Unitaid has granted a no-cost extension

and released USD 0.4 million to be spent after the scheduled project closure.171

There was an overspend on output 5 on coordinated project learning. Final expenditures were

1.7 million, compared to a revised budget of 1.5 million.

For the other outputs (output 2 on increasing provider and consumer demand and output 3 on

improving the quality of case management) and for cross-cutting staff costs and indirect costs,

actual expenditures were closer to budget, deviating from the final revised budget by 10% or less.

Output 6 (on longer-term change leveraging the systems and evidence generated during the

grant) had no separate budget or expenditure lines.

Figure 4. Planned expenditure and actual expenditure172,173

To improve project management in future, grantees should ensure that market information is

comprehensive and up-to-date to ensure optimal decisions and to inform any mid-course

corrections. This can in turn improve grantees’ ability to execute against plans and to warn donors

where this is not possible.

5. MONITORING AND REPORTING

171

Unitaid approved WHO’s separate no-cost extension request specific to output 4. 172

There was no separate budget line or expenditure reported for Output 6 on longer-term systems change, which was covered through other outputs. 173

Expenditure data is from June 2016 consolidated expenditure document, which reports expenditure to 11 April 2016. Budget data is from the final consolidated realigned budget as shown in March 2016 award amendment document.

3

6.4

1.9

Staff costs

Output 4Roadmap development

Output 2Increased demand

Output 3Increased quality of

case management

6.0

2.4

6.4

Output 1Increased availability

1.7

0.1

0.4

2.6

1.7

Output 5Coordinated learning

1.7

6.8

Indirect costs

0.0

1.8

Output 6Longer-term change

0.21.5

89%

94%

97%

30%

113%

100%

USD millionsActual expenditure as proportion of budgeted expenditure

Budgeted expenditure and actual expenditure

OverspendActual expenditure (reported April 2016) Budgeted expenditure (budgeted October 2015)

n/a

n/a


45

Grantees completed routine monitoring and evaluation, with some challenges. Across all

countries, monitoring tools and systems were developed for performance management and

learning. Tools and systems included household surveys,174 interviews with customers, use of

mystery shoppers, and analysis of routine data. Research activities were completed according to

schedule in Kenya and Uganda.175 In Madagascar, Nigeria and Tanzania, the limited capacity of local

boards led to delays in the ethical approval required for surveys. This constrained data collection,

especially at early stages. As a result, some baseline surveys were delayed and in Tanzania no

baseline survey was conducted. As discussed above, the surveys resulted in very small sample sizes

for some impact indicators, particularly for children under five.

The extensive use of surveys posed practical constraints to grantees. In Madagascar and Tanzania,

it was difficult for grantees to manage the external companies that delivered the surveys. These

external companies in turn had challenges in conducting the required interviews. The household

surveys that were called for in the original monitoring and evaluation plan were not conducted due

to practical constraints. PSI suggests investigating the use of routine data rather than survey data in

future.

Inventory data was not systematically gathered on a project-wide basis until June 2015. PSI

developed a reporting tool for PSI countries and MC countries in June 2015 at Unitaid’s request.

Previously, PSI and MC had been tracking stock separately since 2014 according to their own

requirements. In addition, neither grantee monitored distribution of mRDT kits down to the level of

individual sales. In Nigeria and Uganda, the use of barcode technology allowed tracking at the level

of individual kits but MC’s reporting was dependent on providers’ compliance.

Grantees can improve their ability to manage the research process. Grantees should ensure

alignment between feasible indicators and the reporting structure they are held to. Some country

offices can improve their ability to manage external actors that affect the research process. This may

include investigating non-domestic providers to ensure accurate and timely results.

Grantees submitted detailed semi-annual procurement, financial and programmatic reports

although there were some inaccuracies. Deadlines for programmatic and financial reports were

met. The 2015 annual report had some misallocations, and in 2014 MC was not able to provide

documents supporting their statements to auditors, which extended the audit closing period to

seven months, in contrast to Unitaid’s expectations of two to three months. Moreover, the exact

reporting format for some indicators was not agreed upon by Unitaid and grantees, meaning there is

some inconsistency in reporting.

Unitaid felt that grantees did not keep Unitaid sufficiently abreast of risks and challenges. Unitaid

found that PSI and MC’s routine project reporting contained mostly project successes, and did not

sufficiently identify risks, emerging problems, or mitigation strategies. As a result, Unitaid perceived

that grantees did not signal sufficiently the slow uptake in mRDT sales and the difficult market

context in Nigeria. Unitaid did not fully anticipate these challenges and did not appreciate the extent

of the difficulties this posed for the project until it conducted joint visits (with PSI) to Uganda, Kenya,

and Nigeria in mid-2015. Unitaid and grantees also had different perceptions of the risks that the

project posed. In particular, Unitaid considered overstocking as a financial loss to the project, while

174

Use of household interviews was scaled back in favour of exit interviews, at grantees’ request. 175

Indicator O5.2


46

MC and PSI did not. MC considered stock to be the property of the distributors, and considered

them liable for losses and overstocking, whereas Unitaid considered MC accountable.

Grantees should proactively share information on risks and challenges, and the implications this

may have for the project. Unitaid may want to enhance or modify its formal processes. For example,

if challenges are foreseeable, short update reports focusing on risks, challenges and mitigation plans

might be required every three months. Unitaid may also want to emphasise less formal interactions

to engage in constructive and frank collaboration. Grantees can also improve the accuracy and

timeliness of their operational reporting. This includes greater standardisation of operational data

across grantees that is not included in logframe requirements.


47

4. CONSIDERATIONS ON THE NO-COST EXTENSION

Unitaid decided not to accept grantees’ request to extend operations by six months.176 Unitaid

assessed this request based on its perceptions of: project implementation to date; results obtained,

especially in terms of mRDT distribution; and feasibility of delivering the activities described in the

extension application

Unitaid’s denied grantees’ no-cost extension request for three reasons. First, grantees’ progress in

distributing mRDTs was slower than anticipated, which decreased the likelihood of achieving the

objectives set out in the extension application. Second, Nigeria was included in the no-cost extension

application despite all the challenges in the country.177 Third, Unitaid judged that grantees had not

been sufficiently frank about emerging challenges and that risk mitigation plans were insufficient.

Not extending the timeline may have reduced grantees’ ability to achieve some of the project’s

goals. As noted in section 3 on operations, the project had three implicit objectives. Below we

describe the status of each goal by the end of the project and discuss if the no-cost extension could

have further enhanced its achievement.

i. Prove the concept of interventions in private-sector mRDT markets

The project proved the concept of catalysing private-sector mRDT markets. The pilots were

intended to prove if and how a set of interventions could build mRDT demand and supply, and what

interventions were effective in which contexts (and which were not). Despite challenges, overall

proof of concept was established and a range of learnings on the challenges and success factors

were documented, including those highlighted in section 2g. Moreover, the pilots generated insights

into the effectiveness of two different models (bundles and simple commodity distribution). In some

countries, the pilot phase generated evidence that interventions could successfully shape market

conditions, and (potentially equally valuably) the pilot also showed that in other contexts and

countries intervention is more challenging, and potentially requires customised approaches.

Granting the no-cost extension would not have fundamentally changed the outcomes under this

objective, although might have enriched the learnings with further details and insights.

ii. Support the development of sustainable private-sector mRDT markets in target countries

The implementation of pilots was partially on track in most countries and some pilot targets had

been achieved. As discussed in the “findings on programmes” section, grantees delivered a range of

outputs during the pilot phase, including quality assurance of kits, training of providers, and

consumer marketing. Performance against targets was mixed at the pilot stage. Most countries met

certain targets – in supply (including the number of private outlets enrolled178), demand (including

176

As discussed in the introduction section, the first implementation start date was April 2014 (in Kenya and Madagascar) and the last September 2014 (in Nigeria), compared to original plans for approximately October 2013. Given the delays at the outset of the project, the time available for implementation was shorter than originally planned. As a result, in April 2015 PSI made a request to Unitaid to extend the duration of the project by six months (the “no-cost extension request”). The no-cost extension would have allowed a scale-up phase of approximately one year in four countries. As PSI and MC waited for Unitaid’s decision, teams in Madagascar, Nigeria, Tanzania and Uganda began to scale up. 177

Grantees report that they included Nigeria in the application because Unitaid had indicated that it would consider applications on a country-by-country basis. 178

In all countries but Nigeria, the target for the number of private outlets enrolled was met during the pilot phase


48

consumer awareness) and quality (including establishment of QA processes). Other targets – such as

consumer demand179 were missed in all countries. Sales of mRDTs were also a significant problem, as

discussed earlier. Granting the no-cost extension would, however, not have improved performance

in pilot implementation as it was not intended for pilot but for scale-up.

The scale-up was only partially implemented and most scale-up targets were not achieved. The

decision not to grant the extension resulted in a scale-up phase of six months compared to the

original 18 months. Towards the end of the project, grantees had scaled up market-building

operations in four countries.180 As discussed in the “findings on programmes” section, however,

grantees did not reach the intended scale and did not reach most scale-up targets. In most

countries, indicators of demand (including consumer awareness) and supply (including the number

of outlets enrolled) were below targets, although quality of treatment remained high.

Despite limited scale up, some markets grew and appear to be sustainable at local scale, the no

cost extension could have increased the scale further. As discussed in the future prospect section,

sustainable local markets were developed to different extents in Kenya, Tanzania and Uganda. It is

possible that a longer intervention could have increased the eventual market size.

In Tanzania, a range of suppliers were serving the private sector market independently before

project closure. However, PSI was still building consumer demand and recruiting new providers.

An extension would likely have increased the size of the market for these actors (and potentially

others).

In Uganda, an extended scaled-up phase could have recruited additional suppliers. An extension

would also have allowed for increased building of consumer and provider demand, increasing

the size of the market developed.

In Kenya, where scale-up was not planned, the additional six months were less likely to be

transformative. These six months would have funded the continuation of activities in the same

county. However, as in Tanzania and Uganda, the size of the market could have been built

further by continued demand building and provider training.

In Madagascar and Nigeria, the proof-of-concept phase did not establish the longer-term

commercial feasibility of the intervention for different reasons. Scaling up would therefore not have

contributed to the development of sustainable mRDT markets.

In Madagascar, an extension was unlikely to have changed the ongoing need for subsidies nor

contributed to the development of a sustainable market. However, extending the project for an

additional six months might have generated more evidence on operating at scale. In turn, this

may have provided the evidence needed for another funder to continue to fund the model.

In Nigeria, the ability of the planned interventions to build consumer and provider demand was

not proven.

iii. Build and disseminate knowledge and tools to inform the development of other mRDT

markets

The project resulted in knowledge and tools that could be used by other actors, including the

roadmap. During the project, grantees generated, codified and disseminated evidence and

179

Measured as indicator O2.2 as the proportion of consumers seeking fever treatment at private outlets who told the exit survey they asked for an mRDT. 180

No scale-up was planned for Kenya. Nigeria began to scale up until it was ordered to scale down by Unitaid.


49

knowledge. As discussed in the future prospect section, multiple outputs have been produced or will

be produced. The major tool for sharing this research was the WHO roadmap. WHO received a

separate extension to finalise the tool.

However, the richness of content could have been further enhanced. As discussed above, the

project generated evidence and learnings from the pilot phase. However, the project generated

limited insights on how to scale up the markets to national level (crucial for the roadmap, which is

intended to support other governments in their market development). The no-cost extension would

have enabled grantees to document more insights on how to scale a local market.181

It is too early to know if the knowledge and tools will inform the development of other mRDT

markets. The identification and sharing of lessons learned were intended to support development of

private-sector mRDT markets elsewhere. It is too soon for new mRDT markets to have been

launched or developed. However, anecdotal insights suggest that actors in the space see value in the

knowledge and tools that were developed and have used some of these to inform decision making.

181

The project has not yet been able to generate insights on the sustainability of markets; as sufficient time has not yet passed. The no-cost extension has no influence on such information.


50

5. VALUE FOR MONEY

The grant has created and will likely continue to create value. As discussed earlier, the project had

three implicit objectives. The project has already achieved valuable outcomes for each objective.

Additional value may result later from the momentum generated during the project. Finally, in time,

other actors may increase the project’s impact by using the tools and evidence generated to

implement private-sector distribution models in other health commodity markets.

The three combined goals should result in the project’s intended impact: increasing access to

affordable high quality mRDTs to improve health. Increased access to and use of affordable, high-

quality mRDTs can be expected to improve health by avoiding unnecessary use of ACTs, thus

reducing the risk of ACT stock-outs and of resistance from overuse of ACTs. It should be noted that

our assessment of project value is largely based on outcome data (such as the number of mRDTs

distributed) rather than health impact data. The calculation of public health impact would require

additional datasets (e.g. stock-out data) that were not available at the time of the evaluation and

additional epidemiological modelling.

The next section discusses: (1) the value and impact achieved to date, (2) the value and impact that

is likely to materialise in the medium term due to the momentum built, and (3) the value and impact

that might materialise in the long term.

1. VALUE AND IMPACT ACHIEVED TO DATE

The project has proven the concept, resulted in some sustainable markets at sub-national scale,

and generated learnings and materials that are starting to be used. Across the five countries, 2,872

outlets have been signed up and an estimated 1.3 to 1.6 million mRDTs have been distributed.

Market actors are continuing to supply the private-sector mRDT market in three countries. Actors

believe that the private sector can safely handle mRDTs (given the right conditions) and some

governments are considering further regulatory changes. Moreover, significant learnings on what

works and does not work have been generated (see section 2g on learnings). Moreover, WHO is

developing a guide for governments that brings together lessons learned, and an additional 14

knowledge outputs have been developed.182

In terms of impact, the project funded the distribution of mRDTs to private-sector customers who

might not otherwise have had access to diagnostics. Approximately 1.3 million to 1.6 million mRDTs

were sold to consumers.183 This compares with an initial project plan of 4.9 million mRDTs to be

distributed to private outlets.184

The distribution of these mRDTs has likely contributed to public health gains in project areas. The

majority of fever cases diagnosed with mRDTs were negative for malaria, according to project

182

In addition to the roadmap, knowledge resources developed include (i) a chapter for the Springer Encyclopaedia on malaria diagnosis in the private sector, (ii) a set of policy papers with an overview of each of the case study countries, (iii) two qualitative studies on how mRDTs are used in practice in Madagascar and Uganda by the Johns Hopkins School of Public Health team, which may be adapted for publication in a peer reviewed journal, (iv) 9 conference presentations on lessons learned, (v) a manuscript on quality control of mRDT accessories, and (vi) an article drawing on evidence gathered for the project, ‘Building a healthy and sustainable market for malaria rapid diagnostic tests’. This article was published in the WHO Bulletin. 183

This is quantified above in our estimations of reach and the value of stock distributed. 184

End-of-project target for indicator P1 for all five countries and all years.


51

reporting.185 Given that an estimated 1.3 to 1.6 million mRDTs were distributed, the project likely

averted hundreds of thousands of cases where non-malarial fevers would have been treated with

antimalarials.186,187 As a result, unnecessary use of ACTs was reduced in project areas. This is likely to

lead to reduced stock-outs of ACTs and therefore to improved health outcomes for malaria patients

who would otherwise not have had access to ACTs.188

The cost of achieving this impact is approximately USD 11 to 14 per mRDT distributed. Expenditures

relevant to this stage were USD 18.0 million.189,190 This implies total expenditures of USD 6.10 per mRDT

distributed to suppliers or retailers, or USD 11.40–13.80 per mRDT sold to a consumer during the

project.191,192 This figure varies greatly by country, as illustrated in Figure 5. This cost of USD 11.40–13.80

compares with an implied cost of USD 6.79 per mRDT sold based on the originally planned procurement

volumes and budget.193,194

185

The proportion of fever cases diagnosed using mRDTs that were negative for malaria was 81% in Madagascar, 58% in Kenya and 69% in Tanzania, according to PSI caseload data 186

Comparing the proportion of fever cases diagnosed using mRDTs that were negative (footnote 176) to the estimated mRDT sales suggests 260,000 – 330,000 averted ACT treatments in Kenya, 210,000 – 420,000 Madagascar, and approximately 150,000 in Tanzania. A minimum estimate for the number of ACT uses in negative cases averted is therefore 620,000. Data on negative mRDT results were not available for Nigeria and Uganda. 187

This assumes that no fever cases diagnosed with mRDTs would otherwise have been diagnosed. In fact, mRDTs displaced microscopic diagnosis to some extent. 188

Project reporting did not include data that would allow the proportion of malaria patients affected by stock-outs to be estimated. 189

Cost is equal to expenditure against outputs 1-3 (on building demand, building supply and increasing quality), output 5 on codification and dissemination of learning from the proof of concept) and staff expenditure (which is not available by output and therefore includes the cost of staff working on activities not relevant to supporting mRDT distribution). Combined, these expenditures equal USD 18.0 million. 190

This expenditure figure is net of approximately USD 97,482 of project income from sales. This income funded additional programme activity. 191

According to project data on correct management of negative malaria cases, not all mRDT diagnoses resulted in a correct treatment decision. Hence the number is slightly higher per mRDT that led to improved treatment. 192

The unit used in this analysis, mRDTs sold, is the basis for the proxy used to assess mRDT uptake in section 2e above. It is not possible to model value for money in terms of uptake of ACTs. This is because data on the percentage of mRDT sales that lead to malaria diagnoses and ACT treatment is not consistently available across the five countries or across time. 193

This is based on the original budget of USD 33.4 million for outputs 1-3 and 5 and staff costs, and the 4.925 million mRDTs in original procurement plans. 194

This assumes that all mRDTs procured were intended to be sold during the project, which was never an explicit expectation.


52

Figure 5. Project expenditure per mRDT sold, project period, USD

VALUE AND IMPACT IN THE MEDIUM TERM

The project has also built momentum which will extend the value generated across all three goals.

For example, the proof of concept has triggered an interest in mRDTs among other donors, as

described in section 2f on future market prospects. The markets created might continue at current

level or even increase (with support from additional actors).195 In Tanzania, PSI’s mid-term market

research established that there was a landscape of mRDT suppliers that could operate

independently. It appears that independent supply also had been built before project closure in

Kenya and Uganda. Moreover, the project has strengthened systems that will allow the market to be

maintained, including professional oversight systems. In time, additional material, including the

roadmap, will be developed and shared. It is likely that other actors will continue to use knowledge

materials and tools developed to inform decision making.

In terms of impact, if recent sales volumes are maintained in Kenya, Tanzania and Uganda, future

sales of mRDTs in the private-sector market could increase, ensuring longer-term availability of

mRDTs to consumers.196 Assuming that monthly sales of mRDTs continue at the same rate as during

the last six months of the project, the volume of annual sales in Kenya, Uganda and Tanzania could

be approximately 1.8 million.197 At this rate, the within-project goal of 10.7 million mRDT uses could

195

The grant attracted new suppliers and outlets to the market, built demand, and changed the policy environment to allow new groups of private facilities to operate. New suppliers attracted include three wholesalers and five distributors in Kenya, three wholesalers and ten distributors in Tanzania, and two distributors in Uganda. 196

Average monthly sales over the last six months were approximately 35,000 mRDTs in Kenya, 48,000 in Tanzania and 70,000 in Uganda according to sales data. 197

For the last six months of the project, monthly sales totalled 35,000 in Kenya, 48,000 in Tanzania and 70,000 in Uganda – according to sales data. This would imply annual sales of 420,000 in Kenya, 576,000 in Tanzania and 840,000 in Uganda.

2

4.96

11.05

15.88

3.85

19.93

5.78

30.66

UgandaTanzaniaMadagascar NigeriaKenya

NB: Kenya and Madagascar rely on modelled estimates of sales data. Ranges rather than precise estimates are therefore shown.

Project expenditure per RDT sold by country, project periodUSD


53

be reached in approximately six years. These sales would also bring public health gains by averting

ACT stock-outs and resistance.198

Over the coming five years, expenditures per mRDT distributed would fall as the number of sales

to consumers in Kenya, Tanzania and Uganda increases. Including these additional sales in the

definition of mRDTs distributed due to the project would reduce the cost of the project per mRDT

sold. Nigeria and Madagascar are not included due to the assessment that grant activities have not

generated sustainable markets there. In these countries, we assume that costs per mRDT would

remain the same as costs to date (Figure 5) given the probability of reversal if no other market

shaping initiative occurs.199

Figure 6. Project expenditure per mRDT sold, project period and subsequent five years, USD200

VALUE AND IMPACT IN THE LONG TERM

In time, the project may also inform the development of private-sector mRDT markets in other

countries. The project was designed to make significant contributions to the sector’s understanding

of private-sector mRDT markets. The most fundamental addition to the knowledge base is likely to

be the body of evidence on how to generate market interest in and engagement with mRDTs. The

evidence generated has also increased regulators’ and donors’ confidence in private providers’

capabilities to use mRDTs in the project countries.201 The other forms of evidence generated may be

198

The contribution of mRDT sales to these dimensions cannot be calculated due to the absence of data on negative malaria diagnoses in Uganda (in addition to the gap on epidemiological data for all countries). 199

They have not been included because of the assessment that the project did not foster sustainable markets in those countries but require additional donor support. It is therefore not possible to extrapolate future sales and attribute them to the project. 200

Cost is equal to total project expenditure per country against outputs 1-3 (on building demand, building supply and increasing quality), output 5 on codification and dissemination of learning from the proof of concept) and staff expenditure (which is not available by output and therefore includes the cost of staff working on activities not relevant to supporting mRDT distribution). 201

Specifically confirmed for Uganda and Madagascar (possible also for other countries). In addition, CHAI and USAID note that this this project has had a role in proving the private-sector mRDT market, including providers' ability to administer diagnostics safely.

4

0.62

0.76

0.53

Kenya TanzaniaMadagascar Nigeria Uganda

n/a n/a

Project expenditure per RDT sold by country, project period and subsequent five yearsUSD


54

less impactful than envisaged in supporting market actors and shapers, such as governments and

suppliers, to engage in market development. The WHO roadmap was intended to draw together the

evidence generated through the project to guide governments in building private sector mRDT

markets. However, some interviewees were sceptical about the extent to which the limited evidence

on scale-up will meet the needs of the roadmap’s audience. Additional research materials have been

planned but not yet disseminated. Interviewees were therefore not able to assess how the evidence

base built might contribute to future market building.

The research has also generated knowledge and processes that will improve the effectiveness of

grantees, Unitaid, and potentially the sector. As mentioned above, PSI has introduced new

knowledge codification systems that it expects will improve its ability to learn and improve during

other projects. MC has gained and shared learnings on market shaping through “bundling”.

Finally, the project may generate interest to experiment with and potentially expand private

sector distribution in other health-commodity markets, building on the tools and concepts

developed for mRDTs.


55

6. CONCLUDING THOUGHTS

This project was a bold and forward-leaning attempt to tackle a public health challenge by

reshaping private-sector markets. The lack of appropriate malaria diagnosis in the private sector is a

major issue plaguing the global fight against malaria, and new and innovative approaches to tackle

this problem were and are needed. Unitaid and its grantees should be lauded for their commitment

to explore and experiment, their ambition to tackle five complex and diverse markets, their

willingness to look at novel and innovative approaches, and their willingness to take risks.

System intervention is a cycle of trial, error and adjustment. The project achieved several, but not

all, of its stated goals and targets. In some instances, these challenges may have been preventable

by better planning and foresight; in others, this was not possible. We believe that, especially when

intervening in complex systems, it is a fundamental skill to manage feedback loops (observe-

interpret-decide-act) in a timely manner. We hope that the project outcomes, the reflections shared

by Unitaid and its grantees, and the observations in this evaluation will help to further increase the

speed and accuracy of these loops in future projects.

Experimentation and innovation includes learning about what works and about what does not

work. One of the implicit objectives of the project was to understand whether and where the private

sector distribution of mRDT might be successful, and which interventions and approaches work (or

do not work) in different contexts. Given the diverse and complex contexts in which the project was

implemented, certain interventions are inevitably less successful than others. The learnings on what

does not work are in our opinion a valuable outcome of the project, and should be analysed and

shared. Moreover, the project generated a wealth of lessons and insights on the managerial and

organisational requirements on how funders and implementers should plan, collaborate and

manage on these types of innovative projects.

The value of this project will be further augmented by disseminating the lessons learned. We

believe that the analysis of this innovative project, and the sharing of the lessons learned, will be of

great value to many stakeholders in the global health community. We hope that this evaluation

makes a small contribution to that wider goal.


56

ANNEX 1. THEORY OF CHANGE

2

Dalberg understanding of project theory of change

Component included in project logframe

1

Outputs

Purpose(Outcome)

Goal(Impact)

Increase the use of quality-assured RDT results to improve treatment decisions of people with fevers in private-sector facilities of beneficiary malaria-endemic countries

Increased uptake of quality-assured RDTs in private-sector markets

Development and dissemination of a "roadmap" for public-private engagement in malaria case management

1.Systematic reviews of existing national policies and regulatory frameworks that allow for the widespread availability RDTs private sector2.Convene and regularly interact with key stakeholders to advocate for national policies and frameworks that support widespread availability of RDTs in the private sector3.Draw from project experience and systematic reviews to develop WHO recommendations (a "roadmap") for public-private engagement on RDT use in the private sector

1.Convene regular partner meetings and conferences2.Conduct formative market research3.Develop and maintain routine monitoring that captures output level indicators4.Conduct research capturing project outcomes and impact

1.Disseminate project learning and key considerations widely

Affordable, quality-assured RDTs continue to be made available in the private sector by leveraging the systems, learning and evidence base generated

Activities

1.Develop policies, recommendations and guidance for RDT quality assurance through public-private sector collaboration2.Develop and execute country-specific RDT procurement plans3.Develop and execute country-specific plans for RDT quality control in the private sector4.Transfer/adapt quality control systems to incorporate new tools as they become available5.Develop a system for providers and consumers to recognize quality assured RDTs in the private sector6.Develop a system that leverages the public sector to independently monitor private sector providers' RDT quality and user proficiency

Increased availability of affordable, quality-controlled RDTs in the private sector markets

1Increased provider and consumer informed demand for RDTs in private-sector markets

1.Conduct consumer/provider research2.Develop country-specific marketing plans for the pilots3.Develop pricing strategies that consider provider incentives and consumer willingness to pay4.Explore different packaging options5.Develop and implement behaviour change communications6.Create awareness about authentic quality-assured RDTs

2

1.Develop and test treatment algorithms for appropriate case management of malaria2.Train private providers3.Develop and publish private provider standards of practice for malaria case management4.Develop and execute medical detailing/supervision plans

Improved quality of malaria case management among private providers

3 4Coordinated project learning on the introduction of RDTs in the private sector

5 6

Building private-sector mRDT markets in target countries Supporting development of private-sector mRDT markets in other countries

Supply creationPolicy and

regulatory changeDemand creation


57

ANNEX 2. ASSESSMENT OF LOGFRAME

We have assessed the logframe along six dimensions: relevance, attainability, logic,

comprehensiveness, measurability and timeliness. Overall, our assessment is that the logframe

included clear and specific indicators for most variables, time-bound targets, and measurable

indicators as well as a clear strategy for monitoring and evaluating achievement of each target. The

logframe could have been improved by: ensuring that indicators were relevant to the objectives of

the grant and of Unitaid, including additional relevant factors to increase comprehensiveness,

articulating more clearly the logic between outputs and expected outcomes, considering attainability

of targets, and developing specific targets for every indicator. Below we detail each point further.

Relevance: The KPIs collected did not fully align with overall grant goals. At the goal level,

there was no KPI for market catalysis such as number of wholesalers purchasing mRDTs for

distribution in-country. The impact indicator on quality of fever treatments did not address all

dimensions of fever treatment; it did not include the proportion of total fever cases receiving

ACTs as part of treatment, whether mRDTs were quality assured, or information on how

negative cases were treated.

Attainability: Targets did not take account of large variances across countries. Targets at the

goal level and at the output level were mostly identical across all countries. Goal and outcome

targets were not grounded in the baseline results and were not updated to reflect them. For

instance, the baseline results for indicator G1.2 met or exceeded the end-of-project targets in all

countries but Madagascar. Target levels were not adjusted to reflect the specific circumstances

in each country, such as the prior relationships with distributors in Madagascar and Tanzania, or

the more challenging market context in Nigeria.

Logic: The linkages between project activities and outputs and behaviour changes among

other actors were not all clear. In particular, output 6 was designed to measure impact achieved

through changing systems, and developing learning and evidence that could be used by the

broader sector. Because this output tracks change that depends on planned project outputs and

the choices of other actors, it could have been considered as an outcome. Indicators under

output 6 relate to other work streams and could easily have been included under other outputs.

For example, indicator O6.2 on caregivers requesting febrile kits measures demand and could

therefore have been included under output 2. Indicator O6.3 on research dissemination could

have been included under output 5 on learning. This reflects the great difficulty of designing a

logical framework for assessing catalytic impact.

Comprehensiveness: The theory of change addressed a wide range of activities and outputs

relevant to a markets for the poor (M4P) approach but did not prioritise specific activities and

neglected some factors that are important to ensure quality of care.

The theory of change was ambitious in scope and addressed a range of market failures. In each

country, objectives included building demand, improving supply, and influencing the policy

environment. Activities included marketing to consumers and providers, procurement and

managing distribution chains, QA activities, engaging with regulators, and ongoing research and

learning.

The reporting framework did not include some important factors such as the availability of

ACTs. A key assumption in the theory of change was that improved management of negative

fever cases would improve ACT availability. There was no KPI on ACT availability. Nevertheless,


58

project plans were amended mid-term to procure ACTs to ensure availability in Kenya and

Madagascar.

Timeliness: Three years may not have been a realistic timeline for the ambitious objectives

and activities planned. Market development is a long-term process and, as noted above, three

years may not have been long enough to meaningfully engage with these markets. In addition,

this short timeframe increased the effects of project delays. Delays early on significantly reduced

the project timeline and the feasibility of the targets.

Measurability: Targets were measurable, and each had its own measurement strategy, but not

all targets were in fact measured and reported as planned. At goal and outcome level,

achievement of percentage targets was measured using a variety of pre-agreed methods such as

population surveys and client exit interviews. This ensured a degree of objectivity and

comparability between counties and over time. However, many absolute targets were not

measured or modelled. From the outset, grantees did not report against these absolute targets

due to practical difficulties. This suggests low feasibility of measuring absolute targets. At

output level, the agreed measurement strategy for most indicators was also objective and

comparable. The challenges in gathering data and evidence are covered in section 3 on

operations.

In future, Unitaid and grantees can also ensure logframes include indicators that provide more

insight on system-level change during implementation. The indicators measured focused on

grantees’ activities rather than on system-level change. A stronger logframe could provide a better

understanding of the broader system by assessing dynamics between consumers, providers and

suppliers. Such insights would require an element of market research similar to the landscape

assessment PSI Tanzania completed, which assessed players in the market and the role PSI could

take to complement them. As in Tanzania, such analysis would help guide decisions on when and

how to participate in the market and on other mid-course corrections. In the table at the end of this

section we provide some suggestions of additional indicators that would improve Unitaid’s and

grantees’ ability to understand impact and make mid-course corrections.

Similarly, in the future, Unitaid can also support measurement of longer-term, catalytic change

through promoting post-project evaluation. Unitaid could set up structures that allow it to monitor

and evaluate its catalytic effect after the project has finished. Some relevant market data is already

being collected through the Unitaid-funded ACTWatch programme (such as the proportion of outlets

stocking mRDTs). Additional data could be gathered through a one-off landscape review after

project closure. In future grants, Unitaid can ensure that key grantees and actors along the supply

chain have contractual obligations to share the necessary information (on sales volumes at

aggregate level, for instance).


59

Table 3. Illustrative market development indicators

Topic Indicators Source

Players active at each stage

o Number of manufacturers, importers, distributors and wholesalers in the supply market202

Number of outlets providing mRDTs o Qualitative insights on their future

intentions/prospects

Interviews with manufacturers, importers, wholesalers

Commercial relationships between players

Sales volumes between suppliers and providers

Number mRDTs sold to consumers

Aggregate data shared by manufacturers, importers, wholesalers, and/or sample retailers (as possible)

Quality assured mRDTs

o Prevalence of quality-assured mRDTs vs non-quality-assured mRDTs in the market

Data shared by wholesalers and by sample retailers

Consumer satisfaction

o Qualitative accounts of satisfaction/ dissatisfaction of using mRDTs

Interviews with retailers and consumers

How mRDTs are handled

How training is provided and scaled, how waste is managed (could be qualitative)

Interviews with retailers and wholesalers

Other market shaping interventions

Qualitative insights on additional interventions funded by donors/ governments, USD invested, mRDTs moved through these interventions, etc.

Interviews with key donors in country

Policy Status of policy change Interviews with government

Price Price evolution Interviews with actors along the supply chain / data shared by wholesalers and by sample retailers

Key

Indicators from project logframe

o Suggestions for new indicators

202

This information was collected by implementing partners, but not as part of the logframe.


60

ANNEX 3. COUNTRY PROFILES

3A. KENYA

The Unitaid-funded project was implemented in coastal Kwale county. DFID funded a similar market

intervention in two neighbouring counties (Mombasa and Kilifi) and the two projects shared some

resources, including two QA officers. Unlike all other countries, no scale-up phase was planned. The

original plan for the Unitaid-funded project included a target of reaching 100 social franchises and

private pharmacies. However, initial landscaping did not identify that 60 of the 100 outlets identified

were not officially registered. The National Malaria Control Programme only permitted 40 of the

outlets to participate.

In order to allow private outlets to provide diagnostics, a waiver was granted by the Kenyan Medical

Laboratory Technologists and Technicians' Board to allow pharmacists to provide diagnostics tests.

PS Kenya procured 250,000 mRDTs from AccessBio and imported them into Kenya. There was

initially no direct mRDT subsidy, but PSI procured the mRDTs at advantageous wholesale prices pre-

negotiated by CHAI. In addition, rather than working through wholesalers, PSI handled distribution

directly. This acted as an effective subsidy of 100% of distribution costs.

This system was set up based on information on the margins needed by retailers in other market

development projects. A landscape study in 2015 identified that price was a significant constraint on

growth of the mRDT market. Retailers were charging approximately USD 1.00 per mRDT – above the

recommended price – but earning low margins. A direct subsidy was introduced, reducing the retail

price to USD 0.60. Quarterly sales to wholesalers tripled from 9,595 to 26,280 following the

introduction of the subsidy.

Over time, PS Kenya moved away from direct distribution and towards working through existing

market players. PS Kenya also changed its strategy over time to improve availability of ACTs. ACTs

were less easily available in the country between the wind-down of the Affordable Medicines Facility

for malaria and the implementation of DFID and Global Fund subsidies for ACTs. PS Kenya identified

that that this low affordability was a barrier to best-practice malaria case management, and

procured an additional 30,000 ACTs. The original plans to include 500,000 mRDTs were changed

when the slow movement of stock was recognised; only the first of two consignments of 250,000

mRDTs was approved. In order to avoid the expiry of 132,000 mRDTs, Kenya supplied 117,000 kits to

the NMCP, which in turn distributed the kits to private outlets; the cost of these donations was not

recovered.

To date, 52 outlets have been trained.

Future prospects

Due to PS Kenya’s DFID-funded work in contiguous Mombasa and Kilifi counties, some of the market

infrastructure developed in the region is also attributable to DFID funding.

Supply. Quality-assured mRDT kits are now available to retailers on the private market in the coastal

area; three wholesalers and five distributors are currently operating in the market. Distributors

expect further growth and intend to continue supplying mRDTs to providers. Outlets report that the

tests are easily available from suppliers. As of the end of the project, retail prices had fallen to a level

that is affordable to consumers (approximately USD 1.00). It is not clear to what extent suppliers will


61

remain in the market. However, a wholesaler reports that mRDT sales are increasing, resulting in

lower margins being offset by higher volumes.

Demand. A range of stakeholders (including government authorities and wholesalers) report an

increased use of mRDTs. Wholesalers expect the market to continue to grow and intend to stay in

the market. Among consumers, demand has been built to some extent. There is evidence of

consumer awareness and, anecdotally, some consumers have started asking for mRDTs. However,

stakeholders report that consumer demand has not been built as much as it could have been. “There

are gaps. We can do better,” one interviewee from the NMCP said. Supply and price of ACTs remain

erratic, affecting the value proposition of mRDTs relative to presumptive treatment.

Policy change. Prospects for the policy environment are uncertain. As part of a revision of the

national malaria strategy, the government has extended its QA remit to private sector care

providers. The temporary waiver is still in operation. However, there has been no national policy

change. In addition, the NMCP reports that regulatory issues more broadly, including product

specifications, have not yet been addressed. A positive relationship with the NMCP has been built,

which may increase the likelihood of future regulatory changes.

3B. MADAGASCAR

Project summary

PSI Madagascar began a pilot phase in the city of Tamatave in April 2014. PSI procured an initial

consignment of 60,000 single-pack mRDTs from Standard Diagnostics. These were then distributed

to wholesalers, including partners from previous PSI work. Later in 2014 an additional consignment

of one million hospital packs was ordered, although Unitaid approved delivery of only 716,675 due

to concerns about overstocking following slow progress in demand building. Having already paid for

the 283,325 cancelled mRDTs, PSI Madagascar sold these to PSI Angola to recoup the cost.

In 2014, the Ministry of Health introduced a new policy to allow private sector pharmacies to offer

malaria diagnosis, which allowed training of private providers to begin. PSI worked to stimulate

demand from providers through training, and from consumers through behaviour change

campaigns.

A direct subsidy by PSI was used to ensure that mRDT kits were affordable to consumers. The level of

subsidy was approximately half the recommended retail price, with small variations to ensure one

recommended retail price across single and hospital packs (USD 0.47, set with reference to the

highly subsidised price of ACTs). Interviewees across the Malagasy health system reported that this

price is affordable to consumers but that unsubsidised prices would not be.

Nationwide scale-up started in November 2015, around the same time that Unitaid decided not to

grant a no-cost extension. Training and behaviour change therefore happened on a nationwide basis

for approximately six months before the project ended in April 2016. The project has so far enrolled

approximately 450 private outlets. However, sales volumes are lower than anticipated due to low

demand. Up to half of the mRDTs procured may not have been pushed out to retailers.

Future prospects

Supply. Five wholesalers are operational but they have all secured their stock from PSI rather than

from importers or manufacturers. Approximately half of the subsidised stock made available by PSI


62

remains with wholesalers. This subsidised stock will continue to be available to private outlets for

over a year as it will not expire until 2018. PSI and the Direction de Lutte Contre le Paludisme have

applied for further funding from the Global Fund to finance further mRDT subsidies; the outcome of

this application is outstanding.

Demand. Despite increases in supply, demand remains low. Providers are aware of mRDTs but

provider use of mRDTs is not yet systematic. Most consumers – especially those with lower incomes

– do not demand mRDTs. PSI’s ACT project, funded by the PSCM, will take on some demand building

activities including provider training and consumer behaviour change in the future. Ability to pay

remains low for most of the population. The consensus across stakeholders is that, without subsidy,

consumer demand would collapse. Subsidy will be necessary for the medium term. Subsidised ACTs

will remain available in the market through the PSCM and, without subsidised mRDTs, treatment

without diagnosis is likely to remain attractive to consumers – and therefore to providers.

Policy change. Permanent policy changes made during the project are likely to support gains in

availability of mRDTs and quality of kits and of treatment. These include (i) a regulatory text

authorising diagnosis by private providers, (ii) incorporation of the private sector in public sector

systems for assuring quality of provider care, and (iii) a common set of product specifications for

mRDTs in the public and private sectors.

3C. NIGERIA

Project summary

MC’s model in Nigeria gave most responsibility for distribution to the manufacturer. MC procured

2,200,000 "bundles" of single packs of mRDTs and various services required down the supply chain:

stock management, demand creation, provider training, medical detailing, supervision and waste

disposal. MC subsidised the cost of these services by providing a subsidy equal to the difference

between the average cost of the mandated service and its target unit cost. This unit cost was set to

satisfy two targets: (i) a retail price lower than over-the-counter ACTs, and (ii) the margins expected

by players at each step of the supply chain. Single packs were chosen to appeal to operators of small

shops who find it hard to manage the components of the diagnostic (such as swabs, lancets and

alcohol) separately.

There were some favourable policy developments during the project due to the federal

government’s support. Notably, PPMVs were authorised to perform mRDTs, opening up a large

segment of the private provider market.

Procurement was delayed from initial project plans as it took longer than expected to secure

Unitaid’s approval for the new bundled model. Delivery took place in August 2014. There were also

challenges within the MC team due to staff turnover.

Over the course of the project, it became apparent that Nigeria was a more challenging market

environment than initial market research suggested. Significant challenges were: (i) provider

resistance to replacing microscopy with mRDTs, (ii) opposition of professional associations including

lab technicians to allow new facilities to diagnose, (iii) the selection of less economically developed

states, with smaller supplier bases, and (iv) flooding of the local market. This flooding included

mRDTs illicitly taken from the public sector and sold on and a large stock donated to private

providers for free in the three project states by the Global Fund through SFH.


63

The market price for mRDTs remained far lower than the target retail price of USD 1.00, due to the

availability of substitutes. As a result, it was very difficult to build demand. Far fewer outlets than

targeted are operative in the market (868 at project closure, against plans for 3,000). Overstocking

was considerable. Unitaid cancelled delivery of 999,875 mRDTs not yet delivered to Nigeria by one

manufacturer, Premier Medical Corporation).203 This resulted in losses for PMC and a reduction of

total planned stock for Nigeria to 1.2 million. When Unitaid denied the extension in 2015, it ordered

that Nigerian operations further scale down by 85%, effectively ending the project.

Some of the overstock was dealt with by distribution through SUNMAP, an MC project providing

support to some Nigerian states in executing their malaria strategies. 13,450 mRDT kits that were

not distributed in this way or sold have been destroyed.

Future prospects

Supply. The impact of the project on the availability of mRDTs to private-sector suppliers was

limited. PPMVs, a large segment of the private health market, are able to offer mRDTs. The market is

flooded with cheap or free mRDTs and low-quality diagnostic kits. It has been difficult to attract new

wholesalers and distributors to the market for quality-assured mRDTs because of the low margins as

compared to lower-quality substitutes. However, the distributors involved in the MC project – Codix

and SHI – are still supplying mRDTs. The share of quality-assured mRDTs in the market remains low,

according to civil society interviewees.

Demand. It has been challenging to build both consumer and provider demand. The considerable

subsidy did not translate into mRDTs competitive pricing for outlets. This was due to availability of

lower quality diagnostics and free tests given away by the Global Fund. Where demand exists, this is

concentrated in higher income areas, as it was before the project. Provider demand for quality-

assured mRDTs as compared to lower-quality alternatives remains low. Due to the scale-down, the

opportunities to further build demand among consumers or providers are low.

Policy change. The 2015 waiver allowing PPMVs to perform malaria diagnostic tests is still in force.

However, progress in regulation and QA of mRDTs for the private sector remains slow, according to a

Nigerian health NGO.

3D. TANZANIA

Project summary

The start date of the Tanzanian project was pushed back eight-months due to the time needed to

sign a memorandum of understanding with the Ministry of Health. The 215,000 hospital pack mRDTs

PSI procured were delivered in May 2014, compared to a planned start in September 2013.

PSI did not subsidise prices but negotiated a favourable wholesale price by buying in bulk and

passing along savings.

In contrast to other countries where temporary approvals were won, in Tanzania the project was

operational only among formal health care providers in the private sector such as hospitals and

clinics. While the project was initially expected to include accredited drug dispensing outlets

203

Unitaid clarified that these mRDTs had not been approved by Unitaid.


64

(ADDOs), a major segment of the private health care market, authorisation for this segment to

conduct diagnostic testing was not given. This reduced the reach of the project.

During the pilot, PSI implemented provider trainings, behaviour change campaigns and support to

regulators including supervisions of participating facilities. PSI worked with suppliers and facilities

with whom it had prior relationships. PSI Tanzania further built trust and guaranteed demand by

sharing market data and its stock data with suppliers to prove that the market was viable.

No market landscape was conducted at the outset of the project; PSI instead drew on a recent

landscaping conducted by CHAI. The initial PSI landscaping was instead conducted midterm, in April

2015. This market assessment found that suppliers of mRDTs had entered the market and

established relationships with importers and facilities. By 2015, there were additional sources of

quality-assured mRDTs besides PSI. PSI judged that the market could operate independently,

stopped procuring and shifted to a facilitative role where it supported others’ commercial

relationships.

A key challenge identified during the project was illicit sales of mRDTs from the public sector to the

private sector market. Collaboration with government authorities, which ran an employee education

campaign in the public sector and increased monitoring, addressed the problem.

Tanzania began its scale-up to forty districts as Unitaid announced its decision on scale-down. By the

end of the project, PSI had trained 546 private facilities.

Future prospects

Supply. Players operating in the market include three importers and an estimated ten regional

distributors. Distributors report that they perceive market growth and intend to continue supplying

mRDTs to providers.

Demand. Prices rose after directly procured stocks were exhausted. However, the existence of a

broader supply-base has ensured that prices did not rise beyond consumer’s ability to pay. As noted

above, due to regulatory restrictions, the market is currently limited to formal health care facilities

and does not include pharmacies, drug shops or ADDOs.

Policy change. There have not been enduring policy changes in Tanzania as a result of this project. A

separate project currently being executed with CHAI is piloting diagnosis in ADDOs, which may

increase the private provider market in Tanzania.

3E. UGANDA

Project summary

MC used the same model in Uganda as in Nigeria. MC procured 600,000 mRDTs each from two

providers (1.2 million total). PMC and Standard Diagnostics, the same manufacturers as in Nigeria,

supplied the commodities and worked with local importers: Medilink and BDH. These importers in

turn contracted distributors – UHMG, Quality Chemicals and Karuri – to organise the services

mandated by MC: stock management, demand creation, provider training, medical detailing,

supervision and waste disposal. MC retained responsibility for designing the e-learning course used

to train providers and for organising mass marketing.


65

As in Nigeria, procurement approval took longer than anticipated, and approval was given in July

2014.

The project originally operated in Wakiso district outside Kampala. For the project, a waiver was

introduced to allow private providers such as drug stores to conduct diagnostic tests.

A similar model of direct payments to suppliers of mRDT bundles was used as in Nigeria. The level of

subsidy was 82% of recommended retail price. As a result, average retail price remained around the

recommended price of USD 1.00, which consumers were willing to bear and which was less than the

price of ACTs.

MC did not assume control or responsibility for building relationships between distribution partners.

Instead, the manufacturers (and the distributors the manufacturers contracted) built demand as

they chose, using different approaches. This led to a variety of models. Standard Diagnostics paid for

services such as waste management in advance, while PMC paid these in arrears following the final

sale of each commodity to a consumer. PMC’s model appears to have inhibited demand by adding

short-term costs for wholesalers and retailers.

As a result of this overstocking and challenges in building supply, in 2015 Unitaid cancelled the order

of 500,000 mRDTs that PMC had not yet delivered, resulting in revenue losses for PMC and a

reduction of the total stock to 700,000.204

The project scaled up in late 2015, before the no-cost extension was denied. Since that time, the

project has operated in an additional eight districts. The majority of the 700,000 mRDTs procured

have been sold. (13,700 mRDTs that expired before they were sold have been destroyed.) 1,502

outlets were trained and are operating in the market.

Future prospects

Supply. Two of the three distributors have stayed in the market and continue to supply mRDTs to

the private sector: UHMG and Kariuri. A third distributor involved in the project, Quality Chemicals,

has left the mRDT market. In addition to the two multinational manufacturers, SD and PMC, a

Ugandan manufacturer of mRDTs, Astel Pharma, has begun to market to the Ugandan private sector,

selling approximately five percent of its stock directly to private providers. Many outlets intend to

continue to offer mRDTs, and to do so at a price affordable to consumers (despite subsidies ending).

According to supervision data, quality of kits and of handling is high and the National Drug Authority

(NDA) has been satisfied with quality.

Demand. Consumer awareness has been built, and many perceive outlets that offer malaria testing

as of higher quality than those that do not. Retail prices have stayed the same since the end of the

subsidised stock. While demand has been created for the mRDT “bundles” of single packs and

services, these bundles are no longer available on the market. Suppliers report that there has been a

sharp drop-off in demand while providers wait for these to be reintroduced. However, SD does not

plan to manufacture single packs in future as they are insufficiently profitable. Other manufacturers

are also unlikely to start manufacturing them as WHO guidelines currently recommend against use

of single packs. There is therefore a risk that demand has been generated for a different product

than the market can provide for. “They want to wait until we have single packs again. We’ve taught

them to drive a car and now we’re giving them bikes,” said a distributor. The distributor also notes

204

Unitaid clarified that these mRDTs had not been approved by Unitaid.


66

that some providers who were used to getting services such as waste management provided for free

by distributors are dropping out of the market rather than organising these services themselves.

Policy change. Temporary policy change was achieved but the prospects of longer-term

authorisation of private providers to carry out malaria diagnosis are not clear. MC and its in-country

partners are confident that change will occur. Before a permanent authorisation is possible,

however, legislation has to establish the right of the National Drug Authority to make this decision.

The legislation is expected in autumn 2016 and policy change following that.


67

ANNEX 4. EVALUATION METHODOLOGY

The evaluation combined desk research using documents provided by Unitaid and members of the

grantee consortium, interviews with 49 stakeholders, and visits to Madagascar and Uganda.

Dalberg assessed (i) progress against the explicit objectives spelled out in documentation including

project planning and the logframe, (ii) additional objectives understood by project participants and

implicit in project planning – and progress against these, and (iii) explanations for successes and

challenges against each.

Research was undertaken as follows:

• Desk review of documents shared by Unitaid, PSI and MC with Dalberg. The evaluation team

relied on counterparts at grantee organisations to provide:

o Implementation plans.

o Routine documentation, including semi-annual reporting, approvals notices, procurement

reporting, budget sheets, and documents produced as part of the monitoring and evaluation

process.

o Additional important sources, including records of communications between grantees and

Unitaid.

o Learning documentation, codifying the lessons learned and summarising project

achievements.

• Additional desk review. The evaluation team also consulted additional materials, including the

World Malaria Report and policy literature on market development in global health.

• 49 stakeholder interviews. Interviewees were suggested by Unitaid, PSI and MC and spanned

members of the grantee consortium, national authorities and malaria control programmes,

private sector suppliers, private outlets participating in the project, and civil society. Country

teams identified a shortlist of interviewees for each country. For Kenya, Nigeria and Tanzania,

Dalberg conducted three to five phone interviews per country. For Madagascar and Uganda,

Dalberg conducted eight interviews in each country. These in-person interviews included on-site

interviews with wholesalers and private health providers, which offered a first-hand insight into

market conditions. The interviewees are listed in Table 4.

In addition to interviews and document submission, country teams, Unitaid, and PSI and MC steering

committee members provided information ad hoc in response to Dalberg’s questions.

As with any evaluation exercise, there were practical constraints. Due to time limitations, the

number of interviews possible in each country was limited and not all interview targets were

available to speak.


68

Table 4. Interviewee list

Interviewee Country Organisation Title or position

Annet Brenda Mushabe Uganda Allied Health Professionals Council

Quality Assurance Officer

Dr Raharivelo Vololomanitra Madagascar Top Riseau Doctor

Alex Ogwal Uganda CHAI Programme Manager

Theodoor Visser Global CHAI Global Malaria Diagnostics Manage

Christian Nsanzabana & Sandra Incardona

Global FIND Malaria Group

Wellington Oyibo Nigeria College of Medicine, University of Lagos

Professor

Steve Harvey Global Johns Hopkins Assistant Professor

Nina Martin Global Johns Hopkins Researcher

Naomi Woods Global Malaria Consortium Regional Programme Manager

Elizabeth Streat Global Malaria Consortium Senior Public Health Specialist

Edward Idenu Nigeria Malaria Consortium Country Programme Manager

Maxwell Kolawole Nigeria Malaria Consortium Country Director

Godfrey Magumba & Robert Mugerwa

Uganda Malaria Consortium Country Director & Uganda Project Manager

Dr Ebenenzer Baba Global Malaria Consortium Technical Advisor

Dr Hortense Rakotoniriny Madagascar Direction de Lutte Contre le Paludisme

Head of Malaria Case Management Department

Dr Lalanirina Ravony Madagascar Direction de Lutte Contre le Paludisme

Head of Malaria Case Management Department (former)

Dr Kate Kikule Uganda National Drug Authority Head, Inspectorate Department

Dr Dorothy Naisiae (Memusi)

Kenya National Malaria Control Programme

Head

Mgohamwende Fidelis Tanzania National Malaria Control Programme

Focal Person Laboratory

Dr Godwin Ntadom Nigeria National Malaria Eradication Programme

Director, Case Management

Dr Tiana Ravelonarivo Madagascar Ordre National des Pharmaciens

Advisor

Dr Samwel Onditi Kenya PATH Programme Coordinator, Malaria Care

Dr Mbolatiana Raharison Madagascar Somaphar Pharmacist

Nikki Charman Global PSI Project Director

Andrea Cutherell Global PSI QA Technical Advisor

Ann Musuva Kenya PSI Director, Malaria and Child Survival

Desmond Chavasse Global PSI Senior Vice President, Malaria and Child Survival

Stephanie Dolan Global PSI Deputy Project Director

Stephen Poyer Global PSI Research Advisor, Malaria and Child Survival

Gaston Shayo Tanzania PSI -

Edgar Lusaya Tanzania PSI Head, Health Services

Brenda Raphael Mshiu Tanzania PSI Programme Manager Malaria and Child Survival

Dr Rova Ratsimandisa Madagascar PSI Project Coordinator

Dr Mikael Randriamanjaka Madagascar PSI Project Coordinator


69

Ruth Ashibende Kenya Royal Run Clinic Proprietor

Bhavesh Kotecha Kenya Sai Pharmaceuticals Director

Jane Rose Waiswa Uganda St Maama Thereza Pharmacy

Proprietor

Chad Kim and Steve D Uganda Standard Diagnostics Regional Managers

David Katabaire Uganda UHMG Project Assistant

Lorenzo Witherspoon Global Unitaid Procurement Specialist

Ombeni Mwerinde Global Unitaid Monitoring and Evaluation Officer

Ambachew Yohannes Global Unitaid Technical Officer

Ross Leach Global Unitaid Manager, Value for Money

Dr Alexandra Cameron Global Unitaid Manager, Strategy

Matthieu Vittot Global Unitaid Grant Finance Officer

John Cutler Global Unitaid Technical Director (former)

Amy Lin Global USAID Senior Market Access Advisor

Jane Cunningham Global WHO Medical Officer, Quality Assurance

Dr Henintsoa Rabarijaona Madagascar WHO Malaria Control Programme Focal Point


70

ANNEX 5. MODEL METHODOLOGIES

5A. MODEL OF STOCK FLOWS

Stock management data was collected and reported in a variety of formats, reflecting the diversity of models. In order to put stock flows in perspective and to allow aggregation across countries, Dalberg has had to make assumptions based on the available data. The sources for stock data are reported below. Where this is derived from other sources or is estimated, the methodology used is also outlined below. Procured. Stock procured is taken from annual procurement reports and semi-annual project narratives. This includes stock that was procured by partners and subsequently cancelled by Unitaid (in Madagascar, Nigeria and Uganda). Cancelled. This is inferred from the difference between stock procured and stock delivered, and checked against cancellations noted in semi-annual project narratives. Cancellations were due to Unitaid denying permission for delivery of stock procured. Delivered. Stock delivered is taken from semi-annual project narratives. Remaining with PSI or MC. This is inferred from the difference between (i) stock delivered and (ii) stock sold to distributors, wholesalers or retailers and stock donated. Donated. This is taken from data supplied by the PS Kenya country team (where 117,000 mRDTs were donated to the Kenyan NMCP) and UHMG, a distributor in Uganda (which donated 4,000 mRDTs to NGOs). If donations were not reported as such by grantees, they would be counted under stock remaining with PSI or MC, as they were not sold to wholesalers, distributors or retailers. Sold to wholesalers, distributors or retailers. This data was supplied by PSI’s integrated case management team and M&E advisor, who also manage data for MC on this grant. Suppliers are grouped together rather than showing flows between wholesalers and retailers due to the variations in the models used. In Madagascar and Tanzania, data was collected on sales to wholesalers. In Nigeria and Uganda data was collected on sales to retailers. In Kenya, where the model changed, data was collected first on sales to retailers since PSI acted as a distributor and sold directly to retailers, and then later to wholesalers. Remaining with wholesalers, distributors or retailers. This is inferred from the difference between sales to consumers and sales to wholesalers, distributors or retailers. Used for other uses e.g. training. This was taken from data supplied by country teams and includes kits given away for reasons such as training, promotion and invalidity. Sold to consumers. This was modelled by Dalberg, since this data was not reported through routine project processes. Reflecting the differences in models and data availability by country, a different approach was taken for each country. In Kenya and Madagascar, a range is estimated to reflect the uncertainty and imprecision of estimates. In the other three countries, where this estimate is inferred from other stock data, one central estimate is generated,

For Kenya the estimate of consumer sales is modelled based on PSI’s data on case management at participating outlets. From routine reporting on the number of cases tested with mRDTs per month and the number of outlets reporting, the number of cases tested using mRDTs per outlet per month was derived. Three values for cases per outlet per month were used to reflect the


71

uncertainty that comes from using a subset of outlets (e.g. seasonal fluctuations, and whether average number sales per outlet reporting is representative of all outlets). A minimum estimate, central estimate, and maximum estimate were calculated based on the interquartile range of the monthly average. These values were multiplied by the number of participating outlets (which was larger than the number of outlets reporting) to give an estimated range of the total number of mRDTs sold.

For Madagascar, the estimate is based on the PSI Madagascar team’s own assessment that approximately half of the 770,000 stock has been sold and half remain with wholesalers. This was corroborated by interviews with wholesalers. Reflecting the imprecise nature of these qualitative estimates and the range of responses from market players, the estimate uses an upper limit of two thirds of stock sold to consumers, and a lower limit of one third. (Estimates generated from replicating the methodology for Kenya did not match the PSI Madagascar team’s own estimates.)

For Nigeria, the estimate is derived from the difference between the stock sent to wholesalers and the overstock reported in correspondence between Unitaid, PSI and MC on this subject.

For Tanzania, the estimate assumes that, given market developments and the interest of wholesalers and retailers in continuing to supply the market, 0% of stock remains with retailers or wholesalers.

For Uganda, the estimate is derived from the difference between the stock sold to wholesalers and data supplied by MC on stocks remaining with wholesalers. This assumes that no stock remains with retailers, which is unlikely, but is likely to be approximately correct given the interest of wholesalers and retailers in continuing to supply the market.

5B. MODEL OF STOCK VALUE

The estimations of the value of the stock at each stage apply a unit cost to each of the units

estimated in the above analysis.

For Kenya, Madagascar and Tanzania, unit costs are taken from the unit cost paid for mRDT

commodities procured, as reported in annual procurement reports. For Nigeria and Uganda, unit

costs are the price paid to the importer by the wholesaler assumed in MC’s pricing strategy. The unit

cost in the procurement report applies to the whole “bundle” including services, and is not

comparable to other countries.

At procurement, cancellation and delivery, unit cost is reported per consignment. For subsequent

stages, an average unit cost per country is derived from the unit cost of the consignments delivered

and the number of stock per consignment.

5C. CALCULATION OF CONFIDENCE INTERVALS FOR SURVEY DATA AT GOAL AND PURPOSE LEVEL

Confidence intervals are the Wilson binomial confidence interval. This method was selected to

recognise that the sample sizes are small – especially for goal-level indicators – that the population

size is unknown, and that the survey questions are binomial (yes or no).

5D. MODEL OF NUMBER OF MALARIA CASES DIAGNOSED WITH MRDTS AND TREATED WITH ACTS

As outlined in the report, absolute data on the number of malaria diagnoses was not reported during

the project. As a result, section 2e above uses Dalberg’s estimates of mRDT sales as a proxy for

achievement of the purpose-level indicator on mRDT uptake in absolute terms. As mRDT sales are


72

not available by year, it is not possible to apply the annual proportional data from programmatic

reporting to estimate achievement in absolute terms of the goal-level indicator on patients

diagnosed with malaria and treated with ACTs.

Dalberg estimates this using data derived from PSI’s routine case management data. It is therefore

only available for Kenya, Madagascar and Tanzania. For each country, the total number of mRDTs

used is the denominator and the total number of ACTs given to patients who have tested positive

using an mRDT is the numerator. From this, the proportion of mRDT uses that result in a positive

malaria diagnosis and prescription of ACTs is derived.

This proportion is then multiplied by modelled sales data to produce an estimate of total number of

malaria sales that result in positive malaria diagnoses and prescription of ACTs.

This analysis is very approximate. The ability to extrapolate from the subset of outlets reporting is

limited. Deriving the number of treatments using ACTs based on the whole set of mRDT uses

introduces the possibility of error if the rate of positive fever diagnosis is different in the sample

reporting than in the whole population. There are also drawbacks in assuming that one mRDT sale is

equal to one mRDT use in a fever case. This estimate is therefore considered only as an indication of

the order of magnitude of absolute achievements.


73

ANNEX 6. PERFORMANCE AGAINST LOGFRAME INDICATORS

The graphs below chart targets and actual results against the indicators specified in the project logframe. The source is the end of project programmatic report table. The logframe divides these indicators into three levels: goal, purpose, and six outputs. These indicators are shown in table 5 and performance against each is illustrated on the following pages. Table 5. Indicators by logframe component

Goal (Impact): Increase the use of quality-assured RDT results to improve treatment decisions of people with fevers in private sector facilities of beneficiary malaria endemic countries.

Indicator G1.1 Number and % of children under 5 years in project areas testing positive for malaria205 at registered private sector outlets that receive an effective antimalarial treatment by target country

Indicator G1.2 Number and % of children (5 years and over) and adults in project areas testing positive for malaria206 at registered private sector outlets that receive an effective antimalarial treatment by target country

Purpose (Outcome): Increased uptake of quality-assured RDTs in private sector markets

Indicator P1.1 % of Children under 5 seeking fever treatment through private sector outlets received an RDT in the project area of each target country.

Indicator P1.2 % of Children over 5 and adults seeking fever treatment through private sector outlets received an RDT in the project area of each target country.

Output 1: Increased availability of affordable, quality-controlled RDTs in the private sector markets

Indicator O1.1 Number & % of registered private sector outlets in project areas with quality assured RDT brands in stock by target country

Indicator O1.2 Number and % of registered private sector outlets in project areas with no reported stock out of quality assured RDTs lasting more than one week at any time during the past three months by target country

Indicator O1.3 Retail RDT price in project areas by target country

Indicator O1.4 Value (USD) and % (of retail price) of RDT subsidy by target country

Indicator O1.5 Value (USD) of RDTs procured by target country (broken down by type of product, by manufacturer)

Indicator O1.6 Number and % of RDTs procured in line with country-specific procurement plan timeline

Indicator O1.7 Number of countries with adopted recommendations, policies and/or strategic plans for RDT quality assurance spanning public and private sectors

Indicator O1.8 Number of countries using malaria RDT QA systems in the private sector

Output 2: Increased provider and consumer informed demand for RDTs in private sector markets

Indicator O2.1 Number and % of caregivers in project areas who can cite a private provider source of RDTs by target country

Indicator O2.2 Number and % of people surveyed who seek treatment for a fever at a registered private sector outlet ask for an RDT by target country

205

Only people who were tested with an mRDT were included. 206

Only people who were tested with an mRDT were included.


74

Indicator O2.3 Number and % of caregivers in project areas that can recall messages about seeking fever diagnosis in the private sector by target country

Indicator O2.4 Number and % of private providers in project areas that believe that RDTs are a trustworthy diagnostic tool by target country

Output 3: Improved quality of malaria case management among private providers

Indicator O3.1 Number and % of clients in project areas testing negative for malaria at registered private sector outlets managed according to the government recommended treatment algorithm by target country

Indicator O3.2 Number and % of trained/certified providers supervised at least once every six months by target country

Indicator O3.3 % of clients testing positive for malaria managed in private sector facilities in project areas according to the recommended treatment algorithm relative to the frequency of facility supervisory visits by target country

Output 4: Development and dissemination of a "road map" for public-private engagement in malaria case management

Indicator O4.1 Published review of existing national policies and/or regulations and evidence on RDT use in the public and private sector of target countries and relevant additional examples.

Indicator O4.2 Set of recommendations for public-private engagement in malaria case management submitted to the WHO Malaria Policy Advisory Committee (MPAC) for endorsement

Output 5: Coordinated project learning on the introduction of RDTs in the private sector

Indicator O5.1 % of research activities that are completed compared to the research & monitoring plan

Indicator O5.2 # of survey reports disseminated

Output 6: Affordable, quality-assured RDTs continue to be made available in the private sector by leveraging the systems, learning and evidence-base generated through this project.

Indicator O6.1 Number of target countries with public sector mechanisms in place to conduct QA in the private sector

Indicator O6.2 Number and % of caregivers in project areas that request an RDT for febrile cases by target country

Indicator O6.3 Consolidated project learning & key considerations for RDTs in the private sector is disseminated to key stakeholders in each target country


75

Goal

Figure 7. Indicator G1.1

Figure 8. Indicator G1.2

2

G1.1

25%

Round 1

60%

Round 2

79%75%

Round 3

Target Survey results

Percentage of children

Children under five years old in project areas testing positive for malaria* at targeted private sector outlets that receive an effective antimalarial treatment

Notavailable


n = 12 n = 38 n = 1 n = 5 n = 22 n = 14 n = 95 n = 138

20%

0%

Round 3

100%

60%

Round 2

40%

25%

Round 1

20%

100%

50%

Round 3Round 2Round 1

60%

Round 2 Round 3

60%

57%

73%

25%

Round 1

50%

20%

72%

Round 3

60%

Round 2

66%

25%

Round 1n = 1 n = 10n = 9 n = 50

Notavailable

Notavailable

* Only people who were tested with an mRDT were included

3

G1.2

60%

86%

Round 1 Round 3

83%

Round 2

25%


Percentage of people

Children over five years old and adults in project areas testing positive for malaria* at targeted private sector outlets that receive an effective antimalarial treatment

Notavailable


n = 81 n = 109 n = 9 n = 20 n = 61 n = 55 n = 126 n = 713

35%

Round 1

22%

20%25%

22%

60%

Round 2 Round 3 Round 2

52%

69%

52%

60%

Round 1 Round 3 Round 3

60%

64%

Round 2Round 1

25%

69%

Round 3

60%

Round 1

66%

20%

84%

Round 2

73%

25%

n = 9 n = 83n = 160 n = 98

Notavailable

Notavailable



76

Figure 9. Indicator G1 Modelled by Dalberg, per methodology in Annex 5

4

Number of children

600

Target

34,800

86,400

51,000

Sales

27,833


2014 target

Modelled treatments 2015 target

2016 target

35,066

Targets

24,000

260,000

317,000

33,000

Sales Sales

48,256

780,000

Targets

30,000

330,000

420,000

G1

People in project areas testing positive for malaria* at targeted private sector outlets that receive an effective antimalarial treatment


4,30012,048

48,192

192,768

Sales

253,008

Targets Targets

12,480

Sales

5,820

122,220

93,120

23,280


77

Purpose

Figure 10. Indicator P1.1

Figure 11. Indicator P1.2

5

P1.1

17%

4%

30%

26%

20%

48%

Round 2 Round 3Round 1

Survey resultsTarget

Percentage of children

Children under five years old seeking fever treatment through targeted private sector outlets that received an RDT in the project area of each target country

Notavailable


n = 124 n = 208 n = 70 n = 104 n = 205 n = 94 n = 700 n = 273

14%

28%

Round 1 Round 2

30%29%

11%

24%

Round 3

20%

51%

Round 2Round 1

11%

30%

Round 3

51%

Round 1 Round 3

30%

Round 2

24%

38%

22%

30%

51%

30%

Round 1

21%

75%

Round 3Round 2n = 173 n = 70n = 44 n = 256

Notavailable

n =200

6

P1.2

34%

26%

50%

Round 2Round 1

30%

17%

4%

Round 3



Children over five years old and adults seeking fever treatment through targeted private sector outlets that received an RDT in the project area of each target country

Notavailable


n = 408 n = 424 n = 235 n = 339 n = 232 n = 137 n = 466 n = 1,022

28%

Round 2 Round 3

30%

26%

Round 1

17%

29%

14%

Round 3

17%

30%

Round 2Round 1

51%

10%

Round 3

30%

49%

Round 2

41%

24%

Round 1

17%

21%

Round 1 Round 3

72%

30%

Round 2

48%

30%

n = 307 n = 98n = 64 n = 234

Notavailable

n = 293


78

Figure 12. Indicator P1 versus modelled mRDT sales

7

Number of children

SalesTarget

532,600

87,000

80,179

275,600

170,000

P1

Fever cases seeking treatment through targeted private sector outlets that received an RDT in the project area of each target country


2016 target

Modelled sales

2014 target

2015 target

383,907

Targets

790,000

Sales

1,500,000

650,000

60,000

Sales

215,000

Targets

1,400,000

100,000

1,100,000

2,600,000

602,400

2,108,400

Targets

4,518,000

Sales

1,807,200

196,650

605,400

1,571,400

291,000

873,000

Targets Sales

407,400


79

Output 1

Figure 13. Indicator O1.1


8

O1.1


40 40 42

59

98%

52

40%

81%

60%

100%

20162014

60

2015

40%

Number of outlets Percentage of outlets

Targeted private sector outlets in project areas with quality assured RDT brands in stock

Notavailable

Actual results

Target

Actual results

Target

315

89%

2014

38

2015

355

60%210

2016

83%

40%

18

40%

73%

620

2016

74%

60%40%

620

2015

2,840

3,500

74%

2014

140

100%

302

40%

448

2016

672

43%

60%

23

2015

454

40%

100%

40%

2014

201

100%

51

1,500

1,200

1,502 1,502

2015

100%

40%

100%

60%

2014 2016

9

O1.2


30

40 42

60

2016

83%

60%

2015

50

30%

2014

40%

59

98% 96%

Number of outlets Percentage of outlets

Private sector outlets in project areas with no reported stockout of quality assured RDTs lasting more than one week at any time during the past three months

Notavailable

Actual results

Target

Actual results

Target

40%

2014 2016

315

38

14

73%

2015

83%

210

30%

33197%

60%

100%

60%

620

2016

4,260

30%

105

2,840

100%

40%

2015

620

2014

302

100%

454

40%

448

100%

2016

90

2015

19%

672

60%

100%

2014

30%

17

51

40%

100%

2016

1,502

60%

1,800

1,200

100%

2014 2015


80



10

O1.3


2014

1.141.00

2015 2016

Median (IQR) retail RDT price in project areasUSD

Notavailable

1.00

201620152014

2.60

2015

1.00+

20162014

1.59

0.58

2014

0.90

20162015 2015 2016

0.75

2014

0.90Notavailable*

Notavailable

Notavailable

Notavailable

* Reported as USD 0.00 – 1.00 in project logframe reporting+ Reported in surveys as USD 0.00 at PSI franchises and USD 0.42 in other outlets, but not reported in project logframe

11

O1.4


2014 20162015

RDT subsidyPercentage of recommended retail price

Notapplicable

34

6260

20162014 2015

626262

201620152014 2014 2015 2016

848484

20152014 2016

Notapplicable


81

Indicator O1.5

No data reported Figure 17. Indicator O1.6


12

O1.6


000

20162014 20152013

250,000

RDTs delivered in line with country-specific procurement plan timeline

00

20162015

716,675

20142013

60,00000

2014

1,200,125

2015 20162013

0

2016

15,000

200,000

2014 20152013

00

201620152014

700,000

2013

13

O1.7


1

0

20152014 2016

1

0

20162014 2015

1

0

2014 20162015

1

20162014

1

2015

11

201620152014

Notavailable

Notavailable

Notavailable

Notavailable

Notavailable

Number of countries adopting recommendations

Target

Actual results

Countries with adopted recommendations, policies and/or strategic plans for RDT quality assurance spanning public and private sectors


82



14

O1.8


11

2014 20162015

Countries using malaria RDT QA systems in the private sector

1 1

2014 2015 2016

1 1

2014 2015 2016

1 1

2015 20162014

1 1

20152014 2016

Notavailable

Notavailable

Notavailable

Notavailable

Notavailable

Actual results

Target

Number of countries adopting recommendations

15

O1.9


000

2013

138,125

201620152014

00

20152014

22,200

158,878

2013 2016

000

201620152014

1,683,000

2013

0 0 0

2016201520142013

5,400 000

2016

1,002,000

201520142013

Value of RDTs procuredUSD

NB: Tanzania data as reported in project logframe.


83


16

O1.10


000

20162014 20152013

250,000

Number of RDTs procured

00

20162015

716,675

20142013

60,000000

2014

1,200,125

2015 20162013

00

2016

15,000

200,000

2014 20152013

000

201620152014

700,000

2013


84

Output 2



17

O2.1

83%

2016

22%

2015

43%

22%

48%

2014

Percentage of caregivers

77%

42%

201620152014

10%

22%

37%

Caregivers surveyed in project areas who can cite a private provider source of RDTs by target country


54%

201620152014

41%

81%

46%

62%

15%

201620152014

57%

97%

201620152014

9%

19%

Notavailable

Notavailable

Notavailable

Notavailable

Notavailable

NB: Targets were set relative to baselines (30% above baseline for 2014, 35% above baseline for 2015, 70% above baseline for 2016). No baseline took place in Tanzania.

Target

Actual results

18

O2.2

2014

60%

2015 2016

14%

35%

5%

30%

Survey results

Target


2015 2016

69%

29%

48%

2014

39%44%

People who seek treatment for a fever at a targeted private sector outlet asking for an RDT by target country


3%

63%

20162015

2%

38%

2014

33%39%

2014 2016

64%

2015

4%

34%

50%

2014 20162015

52%

Notavailable

Notavailable

Notavailable

NB: Targets were set relative to baselines (30% above baseline for 2014, 35% above baseline for 2015, 70% above baseline for Tanzania for 2016 and 60% for all other countries for 2016). No baseline took place in Tanzania.

Notavailable

Notavailable

Notavailable


85


Indicator O2.4

No data reported Output 3

Figure 25. Indicator 3.1

19

O2.3

70%

35%

16%

2015

34%

2014 2016


2016

17%

70%

35%

2015

29%

2014

Caregivers in project areas that can recall messages about seeking fever diagnosis by target country


35%

2014

22%

2015

70%

22%

2016 2015

70%

2016

6%6%

35%

2014

35%

38%

59%

2014 20162015

70%

Notavailable

Notavailable

Notavailable

Notavailable


Notavailable


86

18

O3.1

2014

99% 99%

2016

40%

2015

92%

30% 30%


Percentage of clients

40%

20162014

82%

96%

2015

30%

Clients testing negative for malaria at targeted private sector outlets managed according to the government recommended treatment algorithm by target country


30%

40%

99%

2014 2015

30%

93%

2016

99%

30%

67%

2014 2016

80%

40%

2015

80%

30%

2016

97%

2015

95%

2014

30% 30%

40%

97%

Notavailable


87



21

O3.2

53%

70%

2016

81% 81%

60%80%

2014 2015


Percentage of providers

80%

20152014 2016

31%

60%

31%

Trained/certified providers supervised at least once every six months by target country


13%

80%

2015 2016

74%

70%60%

2014

85%

201620152014

32%32%

80%

60%

2015

60%

47%

2014

49%

2016

84%

70%

80%

Notavailable

Notavailable

20

O3.3

2014 2015

80%

2016

74%

80%

74%

80%81%


Percentage of outlets

2016

75%

80%

2015

75%

80%

2014

Targeted private sector outlet providers that correctly describe or demonstrate the key steps in the process of conducting and interpreting a rapid diagnostic test for malaria'


80%

94%

2014

46%

80%

80%

2015

68%

2016 2016

80%

2015

72%72%

2014

80%

2016

79%

80%

91%

80%

98%

80%

2014 2015

Notavailable

Notavailable


88

Output 4


Indicator O4.2

No data reported (recommendations not yet made).

Output 5


21

O4.1


1

2014 20162015

WHO conduct reviews of existing national policies and/or regulations and evidence on RDT use in public and private sector

1

201620152014

1

201620152014

1

201620152014

1

20162014 2015

Target Actual results

22

O5.1


1

3

4

2014 20162015

Research activities completed according to plan

3

44

201620152014

2

0

4

201620152014

4

22

3

1

4

2014 20162015

44

3

201620152014



89


Output 6


22

O5.2


1

3

4

2016

2

2014 2015

Survey reports disseminated


2

3

4

2014 2015 2016

0

2

3

201620152014

2

3

2

20162014 2015

2

3

44

20152014 2016

26

O6.1


1 1

201620152014

0

1

20152014 2016

1 1

201620152014

1 1

201620152014

1 1

201620152014

Target countries with public sector mechanisms in place to conduct QA in the private sector

Number of countries with QA mechanisms

Target

Actual results


90



27

O6.2

20162014 2015

28%


6%

2014

21%

20162015

Caregivers in project areas that request an RDT for febrile cases by target country


45%

2016

10%7%

65%

2014 2015

25%

2014 2015

4%

64%

2016

44%

24%

2016

7%1%

2014 2015

Notavailable

Notavailable

Notavailable

Notavailable

NB: Targets were set relative to baselines (20% above baseline for 2014, 40% above baseline for 2015, 60% above baseline for 2016). No baseline took place in Kenya, Tanzania or Uganda.

Notavailable

Notavailable

Target

Actual results

25

O6.3


0

1

0

20162014 2015

Countries in which project learning and key considerations for RDTs in the private sector have been disseminated to key national stakeholders


1

0 0

201620152014

00

1

201620152014

00

1

20152014 2016

0 0

1

201620152014


91

ANNEX 7. ESTIMATIONS OF STOCK FLOWS

Figure 34. Estimated stock flows by value, USD

Source: Dalberg analysis (per methodology specified in Annex 5)

Figure 35. Sales of centrally-procured mRDTs, 2015-16, Units sold per month

Source: PSI stock management tracker

2

Combined By value (USD 000s)

Source: 2015 annual report, Procurement narrative; PSI; Dalberg analysis

Distributed to wholesalers, distributors or retailers

6

1,769

Sold to consumers

Used for other uses

e.g. training

~800–870

Remaining with

wholesalers, distributors or retailers

~900–960

Donated

68

Remaining with PSI or MC

1,874

1,2363,110

Procured Cancelled Delivered

37

Kenya UgandaTanzania NigeriaMadagascar

Estimated unit cost of RDT stocks by stage of supply chain by countryUSD

2

2015-16 sales volumes

NB: Data for before June 2015 is outstanding

Source: RDT stock tracker

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

September 2015

August 2015

February 2016

November 2015

July 2015 March 2016

January 2016

April 2016December 2015

October 2015

Uganda TanzaniaNigeriaMadagascar Kenya

Monthly salesThousands

Documents

Unitaid end of project evaluation: Creating a private ...€¦ · 24/02/2017 · Unitaid end of project evaluation: Creating a private sector market for quality-assured mRDTs –