Theoretical Framework & Practical Challenges in Implementation

Sanjay SaxenaSep 2012

Value for Money 

Need for VfM

• OECD countries deliver $120 billion in aid a year. Yet significant concerns remain regarding the impact of much of this aid.

• Overall around a third of global aid is spent  on TA. One estimate is that half of this aid (US$ 20 billion per year) ‘is wasted'. 

• The Global Fund to Fight AIDS, Tuberculosis and Malaria has spent about $23 billion to save 8.7 million lives over the past decade, but could it have done even more with the same amount of money?

• "Vaccines are one of the most cost‐effective public health interventions and one of the best buys you can get for your bucks. If you are not investing in that, what are you investing in?“

• Donors such as the World Bank and ADB spend heavily on infrastructure – nearly 50% of their portfolio. 

What is VfM about?

• Are we getting the most out of the resources we have? • Are we doing the right things in the right way to achieve the 

greatest impact … and at what cost? • Are we able to demonstrate what we achieve and what more we 

can achieve? 

What is VfM?

DFID “Maximising the impact of UK aid so it makes the most difference to the  poorest people in the world”.

Ditchburn, the Director of DFID VfM Unit –‘The determination to get the most impact for the money we have’.

What is VfM?

The UK’s National Audit Office (NAO)‘The optimal use of  resources to achieve intended outcomes’. 

New Zealand Aid (NZAID) “Achieving the best possible development outcomes over the life of an activity relative to the total cost of managing and resourcing that activity and ensuring that resources are used effectively, economically, and without waste”.

What is VfM?

OECD “The optimum combination of whole‐life cost and quality (or fitness for purpose) to meet the user’s requirement. It can be assessed using the criteria of economy, efficiency and effectiveness”

(Glossary definition ‐ Not official).

What is VfM?

Global Fund “VfM in the health sector is defined as creating and complying with rules or procedures for allocating resources that elicit the production and utilization of the health maximizing mix of health services for the available donor, national and private resources.”

What is VfM?

Definitions

• No clear or universally agreed definition

“VFM is about obtaining the maximum benefit over time with theresources available. It is about achieving the right local balancebetween economy, efficiency and effectiveness, or, spending less,spending well and spending wisely to achieve local priorities...VFM ishigh when there is an optimum balance between all three elements,when costs are relatively low, productivity is high and successfuloutcomes have been achieved.”

• VFM not restricted to the procurement phase alone but in all project programming and delivery 

VfM Conceptual Framework

Successfully achieving the intended 

outcomes from an activity

Minimising the cost of resources 

used for an activity

Maximising output for a given input or minimising 

input for a given output 

Effectiveness

Economy Efficiency

Value for Money and 3Es

Value for Money

VfM Conceptual Framework 

Costs £ Inputs Outputs

Economy Efficiency Effectiveness

Quantitative

Qualitative

Value for Money

Outcomes Impacts

Output: Tells you an activity has taken place and is usually quantitative 

(e.g. number of people vaccinated)

Outcome: The change that occurs as a result of an activity (e.g. reduction in infant mortality rate)

‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ Equity ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐

VfM Considerations

Economy consideration Efficiency consideration

Can we get same or equivalent inputs for less money?

Would using cheaper, different inputs risk the effectiveness, including sustainability?

Would using cheaper inputs risk greater maintenance costs over the life of the 

project?

Can we get the same results while saving on how we manage the activities?

Would making savings to how the project is managed risk reducing effectiveness, 

including sustainbility?

Would making savings on how the project is managed risk incurring other costs?

VfM Conceptual Framework

Key Components Measures Modifiers Contributor 

ShareConfidence 

Levels(Confidence/Assurance)

BenefitMeasures

EffectMeasures

EfficiencyMeasures

EconomyMeasures

Cost Measures

BenefitShare

Cost Share

Context Risk

Assumptions

Effectiveness

Efficiency

Economy

Outcomes

Outputs

Activities

Inputs

Unplanned benefits/ 

dis‐benefits

Unplanned costs/ intangible costs

Benefits

Costs

1. Key Components 

Value for money examines the optimal relationship between costs/resources and benefits/outcomes ‐ delivered through processes that transform inputs through activities to outputs which are necessary and sufficient to trigger outcomes.

– Economy covers aspects such as the procurement of trainers, the cost of hiring room, and the cost of hiring trainers – including the possibility of unit costs compared to benchmarks for similar training available in the country. 

– Efficiency: How many judges are trained? How many trainers did we need? Can a measure of productivity be devised? 

– Effectiveness: Whether judges trained led to improved access to justice? Are there measurable changes at the outcome level? 

2. Measures 

• 3Es measure the relationship between key levels in a hierarchy of programme objectives

• The key elements and considerations (and cautions) of a VFM assessment

3. Modifiers

• Contextual matters, such as it being expensive to run training courses in Afghanistan; Considerations of risk, such as perhaps we may be wrongly assuming that the problem is solved through training. 

• The optimal balance of the 3Es requires the factoring in of context, risk and assumptions which set limits on effectiveness, efficiency and economy. 

• In terms of modifiers, there may also be intangible costs and benefits to factor in which influence judgements on VFM. In addition, at the end of any programme, VFM judgements made at the outset would need to consider not only performance against plan but unplanned costs and benefits.

4. Contributor Share

• There are significant challenges to meet in determining how toattribute costs and benefits when making value for moneyjudgments. This component of the VFM framework is a reminderthat the attribution/contribution questions need to be answeredin any VFM judgement and that assuming a pro‐rata claim onoutcomes based on inputs may be too crude an approach.

• Even if perception surveys show that people feel that the judicialsystem is fairer, how do we know that it was due to ourintervention? How do we consider issues of contribution andattribution?

5. Confidence Levels 

• Data quality plays a significant part in any VFM judgement – this is in part linked to how explicit we are about the assumptions we make about reliability, relevance and robustness of the data sets we used (this relates strongly to indicators) and more explicitly is linked to how sensitive our VFM findings would be to changes in any assumptions made. 

• Bearing the assumptions in mind, a number of options need to be considered with a view to judging them in terms of: – How well they strengthen the ‘key components’ in the framework – To what extent they capture and enhance measures of economy, 

efficiency and effectiveness – Whether they deal with attribution questions robustly – How susceptible they are to poor data quality or associated data 

assumptions – How reliable is the data? 

Money

Environment

People Inputs Outputs

Resources / Investment

Service & Wider Outcomes

Economic

Environmental

Social

Often VfM is understood by

comparing unit costs

Real VfM is achievedby comparing outcomes

with investment 

VfM Conceptual Framework

VfM Isn’t….

• Focusing on Cost Only• Choosing the Cheapest Option• The Easiest thing to Measure• Spending less than Total Budget 

To Summarise….

• Good VFM achieves a balance across the 3 Es;

• Spending less; spending well; spending wisely

• Fourth ‘E’ – Equity  

“Wider social and environmental costs and benefits for which there is no market price also need to be brought into any assessment. They will often be more difficult to assess but are often important and should not be ignored simply because they cannot easily be costed.”

HM Treasury Green Book

Improved RBM

• Definition: which improvements in results based managementand other project management processes are used todemonstrate improved VFM

• Method: Based on audit practice, this option assumes that goodpractice leads to good results. Very similar to quality programmingor ‘effective programming’ approaches.

• Strengths: Very ‘do‐able’: supports current focus on results andencourages strengthening of existing approaches; strong evidencethat improved RBM does lead to improved results.

• Limitations: The focus is on the process rather than the product.Unlikely to be sufficient to satisfy current pressures to attain VFMwithin DFID.

3Es Rating and Weightings Approach

• Definition: Key processes associated with ‐ and measures of ‐Economy, Efficiency and Effectiveness are identified and used torate programmes; and where these processes and measures canbe weighted to reflect their relative importance.

• Method: This enhances Improved RBM Approach by selecting anumber of management criteria and judging performance againsta standard so that programmes can be rated on this basis. Sincecriteria would vary in importance, they might also need to beweighted. Processes that support management as well asmeasurements of the 3Es would be used. The use of weightingscan help overcome commonly‐held assumptions that the VFM issolely about cost‐cutting, with a rebalance towards key aspectssuch as effectiveness.

• Strengths: Utilises an auditing framework; introduces possibilityof benchmarking standards.

• Limitations: Benchmarks not yet developed; contextualdifferences might make comparison difficult.

Trends Analysis:

• Definition: a comparison of progress against indicators is used to measure effectiveness and in which baselines, targets and underlying trends are all taken into consideration. 

• Method: This approach explores the scope for using indicators toestablish a scale against which progress against outcomes can bemeasured and compared (at least against the ‘do‐nothing’ option).Requires relevant and robust indicators, baseline information,monitoring against indicators, and an ability to determinecontribution.

• Strengths: Trends analysis can be used to show a positive contributionof an intervention to a generally downward trend, i.e. that thedownward trend would have been greater without the programme.This can be important in many development contexts, particularlyperhaps in fragile states.

• Limitations: Caution is needed before making cross‐countrycomparisons; requires audit scrutiny to ensure measurementprocedures are adhered to.

Cost Benefit Analysis (or SROI)

• Definition: Outcomes are monetized/financial proxies are introduced in order to make a direct comparison between costs and the financial value of benefits. 

• Method: Fully monetized indicators of benefit are compared to costs; involves the development of financial proxies for all programme benefits. 

• Strengths: This approach provides a simple, comparable read out on the return on investment. 

• Limitations: The approach is extremely sensitive to assumptions and there is a risk that these are manipulated to generate desirable results. – Developing and using criteria to rate and weight programmes – Developing criteria for rating programmes (how they were 

developed) – Rating scale, weighting and scoring programmes against criteria – Table of criteria – Testing a prototype: key points from feedback at VFM Research Day 

Group Exercise

• Do you already look at VfM in your work but use another term, or don’t use a term but still practice it? How do you ensure VfM?

• What can we learn from how value for money is considered in other areas of public sector spending specially in the ‘developed’ countries and indeed from the private sector? 

• Do you think the current focus on VfM is justified? 

Who Drives VfM

Tax Payers

Donors

Beneficiaries 

Donors Accountable to

Tax Payers?

Beneficiaries?

This results in……

• Upward Accountability 

• Lack of Participation

Who Drives VfM

• Current VfM debate is Donor driven • VfM Defined by Donor Agencies • The ‘Value’ in VfM Determined by Donor Agencies 

Who Drives VfM

• DFID decided to focus its bilateral spending on 27 priority countries and close down 16 bilateral programmes by 2016. 

• One of the programmes to be closed down is in Burundi DFID decided that its bilateral aid programme to Burundi offered poor value for money compared with other, larger country programmes. 

“Our aim is to spend every penny of every pound of your money wisely and well. We want to squeeze every last ounce of value from it. We owe you that. And promise you as well that in future, when it comes to international development, we will want to see hard evidence of the impact your money makes. Not just dense and impenetrable budget lines but clear evidence of real effect.”

Secretary of State for In International Development Andrew Mitchell‟s pledge to the taxpayers 

Who Drives VfM 

Poor people are the ultimate beneficiaries of aid, and when evaluating VFM and effectiveness it is essential to consider what poor people want (e.g. Jobs, roads, personal security, dignity)

NHS’s methodological approaches uses a three stage process: 1) What outcomes do patients want ‐ generate a set of development 

priorities through the use of qualitative surveys, focus groups etc(Integrating the priorities of poor people is key)

2) How do people value the different outcomes ‐ identify what combination of outcomes can development practitioners and recipients identify as the most immediate priorities. 

3) What are the costs / benefits of different methods of resource utilization ‐ once priorities are identified it is then possible to compare different ways and combinations of using resources to achieve outcomes.

Practical Challenges

Practical Challenges

Determining ‘Value’ aspect of VfM in a clear and focused  manner, using the Theory of Change

InputsOutputsOutcomes Impact 

Value is adjusted...not by any accurate measure, but by the higgling and bargaining of the market, according to that sort

of rough equality which, though not exact, is sufficient for carrying on the business

of common life.

Practical Challenges 

Measurement ‐ complex, dynamic aspects

‘the most effective development initiatives are the least easily counted and measured, while the ones most easily counted and measured are often the least effective’ 

Andrew Natsios (ex‐head of USAID)

Subjectivity of ‘Value’ ‐ Value means different things to different people. 

Practical Challenges

Attribution How much credit can your initiative take for the outcomes?

• Deadweight: what would have happened anyway?

• Attribution: how much is due to our project, and how much down to other factors?

• Displacement: have we just moved an outcome to / from somewhere else?

• Benefit period: how long does the outcome last, and does the effect ‘drop off’ over time?

Practical Challenges

Capacity ‐ To Monitor Projects and Programmes through a VfM lens

Practical Challenges 

Data & Statistics ‐ availability and collection

Time consuming

Practical Challenges

Implications on InnovationMay land up promoting safe‐‘proven’ techniques

Practical ChallengesExclusion A mechanistic VfM approach can affect aid for:• Fragile states • Conflict zones• Poor & excluded 

• Donors themselves often don’t understand VfM and apply it very mechanistically!

Practical Challenges

Group Exercise ‐ How to get VfM right?

• What are the best practices / principle for ensuring VfM in your projects and programmes?

• What can we do to increase the voice of beneficiaries in holding donors accountable for spending aid intended for them wisely?

• To what extent should we be seeking a co‐ordinated approach amongst donors and partners in how we think about value for money in aid? Or should donors all be working separately given their own domestic demands for accountability and good value in development aid? 

Way Forward

Way Forward

Recipient ownership ‐ Enhance beneficiary participation in determining ‘Value’, which in turn will increase:

– Coverage – Efficiency– Effectiveness– Capacity and Sustainability  

Way Forward

Adopt a flexible approach to avoid risks of exclusion – test and pilot first VfM assessments, ratings, reporting

Way Forward

Enhance Institutional coherence in development and application of VfM

Way Forward

Strengthen Data collection and statistical systems 

Way Forward

Improve capacity to monitor and evaluate VfM

Way Forward

Increase linkage between VfM evaluations to decision making 

Way Forward

Promote research, learning and knowledge management on VfM –build VfM assessments, ratings and reporting into Theory of Change concepts  

VfM in Practice 

Global

Country 

Project

How to apply VfM in practise

How to structure agreements to create stronger incentives for 

value for money

How to verify performance to generate greater incentives and 

accountability for value for money?

How to collect and use cost data for commodities, supply chain and service delivery to leverage value 

for money

How to allocate resources to maximize value for 

money

Allocation Contracts

VerificationCost &

Spending

Programmes that deliver VfM should have…..

Costs that are linked to clear outcomes: Granularity of data reporting and frequency of measurement 

Management systems in place and project design to ensure effective delivery of outcomes and strong risk mitigation 

Project design that prioritises sustainability and capacity building: focus on technical and business capacity building, knowledge transfer and effective handover strategies and procedures 

Project design that looks outwards: complementarities with other programmes and initiatives; an understanding of VfM across all development projects and aid modalities; and ensuring that each project addresses recipient priorities. 

A

Presentation