Upload
lambert-giles-melton
View
233
Download
10
Tags:
Embed Size (px)
Citation preview
Content
• Added value in Structural Funds• PCM / LFA• Theory of Change• Output based subsidies• Outcome mapping• Innovation Pipelines• Conclusions
1
Performance based financing
• There is a tempting idea to skip all the problems of input based financing (paying for effort - inputs and activities) and focus only on delivering results.
• What is important is the result delivered, not the way how it was created. Let‘s directly link financing to outputs/outcomes delivery.
• This part of the seminar investigates this idea.
Structure of presentation
• Introduction• General Output Based Aid model• Case study: Czech Kindergruppe output
based call• Group work• Conclusions
Linking finance to performance - termsFinancing linked to
Outputs Results
Incentives
primarily for
Governments
Service providers / Project promotors
People / Clients / Target groups
PBI - Performance based incentives, P4P – Pay for perf., RBF – Results based financing,PBC – Performance based contracting
CCT - Conditional cash transfers
OBA – Output based aidOBF – Output based financing
PBC – Perf. based
contracting(health)
Vouchers
CoD – Cash on deliveryPforR – Program for results
DLI – Disbursement-linked indicators
Source: Author, based on Performance-Based Financing Toolkit (World Bank)
OBA – focus of this part of the seminar
Key messages to take from the ToC part
• It requires significant expertise to develop a good Theory of Change
• It is a static approach, that assumes the world remains stable for a while
Output Based Aid: A very brief intro
• Output based aid is the contracting out of service provision to a third party with payments made after the delivery of specified outputs.
• It involves formulating a theory of change, but the difference from previous approach is that the ToC is done by the PMO to prepare a blueprint/template for standardized intervention.
• Delivery partners cannot decide on objectives, outcomes and outputs, as these are given.
Output Based Aid: A very brief intro• OBA approach is suitable for „enhancer” PMOs:
For basic, stable, large volume, well-understood services Often used to close the gap between ability to pay and the actual
cost to ensure consumption of certain services at socially optimal level
ToC is used to make clear how output contributes to longer term outcomes
• In Systems thinking words, OBA approach is suitable for interventions with predictable demand with low variability.
Output Based Aid: A very brief intro• As an alternative, enhancer PMOs can also stick to an
input financed approach: Keep ToC simple with a straightforward planned output-outcome
intervention logic
• In both cases, support “Toyota Production System”- like service delivery mindset.
• Again, this is a static approach.
The GPOBA model
• The Global Partnership on Output-Based Aid (GPOBA) is a multidonor trust fund administered by the World Bank. Its purpose is to fund, demonstrate and document OBA approaches to supporting the sustainable delivery of basic services to those least able to afford them.
• Many experience from development countries, mainly used for basic infrastructure (electricity, water supply), but also in health and education sector.
• Universal principles enable OBA use in the ESIF.
Output Based Aid: Six Core Concepts
1. Targeting of subsidy: it is clear to whom, why, and for what the subsidy is provided. End-users are intended to benefit directly.
2. Accountability: by paying only after delivery of pre-agreed and verified outputs, risk is shifted to the service providers who are accountable to both their clients and the donor (PMO).
3. Innovation and efficiency: Focus only on outputs leaves service “solutions” partly up to competition of providers.
4. Using incentives to serve the needed: OBA encourages service providers to serve those they might otherwise disregard.
5. Output verification and monitoring: monitoring of outputs is easier and more precise in an OBA scheme as the payment is made only for independently verified outputs.
6. Sustainability: OBA focuses on affordability for users (pay user fees they can afford and are willing to pay), cost recovery for service providers and future sources of funding.
Output Based Aid: Typical Scheme
Donor/ Government
Independent Verifying Agent
Target GroupService Provider
Implementing Agency
Selection of Agency, Funds
provision
Contracting SP
Services provided
(Fee for service)
Service delivery
verification
Service delivery
confirmation
Output based payment
Output Based Aid: Typical Scheme
EC / Government
? IVA
Target GroupDelivery partners
PMO
OP
Call for projects
Services provided
(Fee for service)
Service delivery
verification
Monitoring report
Payment claim disbursed
Designing OBA Scheme – 8 steps
Source: GPOBAThis can be visualized in a form of Theory of Change!
Designing OBA Scheme A - Determinign the output
What service is to be provided? Define the output units.
Suitable are services that are:
Keep in mind possible perverse effects of narrow definition of the service (gaming, creaming…)
Clearly definable both in terms of quantity and quality
Easy to verify and measure
As close to the desired result as is feasible
Designing OBA Scheme B – Reaching the targer population
How to determine who should have access to the services? More targeting strategies:
Cost – benefit of targeting, nature of the service and other policies in place should be considered.
Geographic targeting
Self-selection targeting
Income testing targeting
Community based targeting
Combination
Designing OBA Scheme C – Choosing a Subsidy Form
Subsidy should fill the gap between the total cost of a service and the user‘s ability to pay the cost.
One-off subsidies
Transitional subsidies
On-going subsidies
Designing OBA Scheme D – Determining Value of the Subsidy
Subsidy = unit cost – ability of user to pay.
Unfortunately, both actual costs and ability to pay is often unknown at OBA design stage.
Ex-ante approximation
Competitive process
Use more sources, verify independently, benchmarking.
Let the market decide, danger of „the winner‘s course“.
Designing OBA Scheme E – Linking Outputs to Subsidy Disbursements
As the subsidy is disbursed after the delivery of outputs, consider the timing of outputs and payment verification and the ability of service providers to pre-finance outputs. Advance payments could be necessary.
Designing OBA Scheme F – Organizing the Institutional Framework
Several questions to ask:Who are the expected Service Providers?Do SP have technical expertise to deliver service?Do SP have sufficient capacity for our policy goals? Are SP private, public, NGO? Are SP ready to take the risks? Is the service subject to governmental regulation?How the results will be verified? (IVA?)
Some answers may require PMO to do additional actions (capacity building, regulatory initiative).
Designing OBA Scheme G – Evaluating and Mitigating the Project RisksRISK MITIGATION
Performance Risk: Output not delivered in agreed terms.
Verification controls.Capacity building before implementation.
Payment Risk: Output delivered, but payment delayed.
Clear disbursement procedures, PMO internal procedures quality.
Demand Risk: Wrong estimate of demand for the service.
Demand studies, awareness rising campaign.
Unit Cost Change Risk: Factors out of SP control.
Careful unit cost calculation and ex-ante evaluation.Mechanisms allowing adjustments.
Collection Risk: Target group not able/willing to pay their part.
Willigness-to-pay surveys
Regulatory Risk: Change in regulation of the service.
Quality of regulation check in design phase. Good communication between the PMO and the regulator.
Designing OBA Scheme H – Organizing Monitoring and Evaluation
• Monitoring and data availability is integral part of the design.
• Available info on costs and outputs enables CBA.• Reporting burden is on Service Provider, verification on
Independent Verifying Agent.• Linking disbursement to monitoring makes inevitable
management attention. • Client satisfaction surveys could serve as quality control.• Evaluation still necessary to confirm relevance and
causal link between output delivery and results.
OBA Suitable when…
Source: GPOBA
Output Based FinancingCase study, part I:Czech Kindergruppe output based call
Linking Finance to Performance with Output Based Financing
The Czech case study:The Problem to be solved• Problem definition:
Too many parents (mostly women) for too long time leave the labour market because of children.
• Consequences:• Parents loosing their competences on the labour market.• Part of skilled labour force is absent on the labour
market.• Higher demands on social system.• Lower income of the national budget.
• Policy objective:More (women) sooner back to work.
The Problem to be solved
• Policy objective:More (women) sooner back to work.
• General solution:Parents have to have the need to re-enter the labour market and the possibility to satisfy this need. Need is affected by
• Economic parameters: Is it economically worth it to start to work (function of salary, work-related costs, taxes and social benefits)
• Motivation for personal development• Societal pressure: „Only bad mothers start to work early“
Possibility is affected by• Affordable child-care services in acceptable quality• Availability of jobs (esp. part time jobs).
The Problem to be solved
• Policy objective:More (women) sooner back to work.
• General solution:Parents have to have the need to re-enter the labour market and the possibility to satisfy this need. Need is affected by
• Economic parameters: Is it economically worth it to start to work (function of salary, work-related costs, taxes and social benefits)
• Motivation for personal development• Societal pressure: „Only bad mothers start to work early“
Possibility is affected by• Affordable child-care services in acceptable quality• Availability of jobs (esp. part time jobs).
Focus of the „Kindergruppe“ intervention
Intervention design
• Main objective To enhance participation in the labour market
by reducing the barriers for parents with small children by supporting creation of employer-based childcare facilities (non-occasional and non-short-term care) – „Kindergruppen“.
Intervention design
• Identified obstacles: Lack of know how on the side of employers. Lack of legal certainty: such facility is possible to run,
however it is not explicitly defined in the Czech legislation and there are divergent attitudes of relevant authorities (hygiene, fire department, building authority).
Economic parameters: full costs at level of 7K – 10K CZK per child/month, when most parent could afford such service for max. 3 – 4K CZK.
Low trust of some parents due to lack of „institutional guarantees of service quality“.
= 27 CZKMedian wage 17K CZK netto
Intervention design• Actions taken
Preparation of ESF Funded output based grant scheme in order to
• Provide employers a blueprint for such a facility• Provide funding for launch and initial phase of operation• Piloting of output based grant scheme in ESF.• Current situation: 65 Kindergruppen supported, first welcomed
children at the beginning of September 2013
• Adopt Kindergruppe legislation in order to• Provide legal certainty• Contribute to solution of economic part of the problem by tax
deduction arrangement for employers• Current situation: Law prepared, legislative process interrupted
by the fall of the government, however the law is still a priority of the new government and should be in force from September 2014.
Linking Finance to Performance with Output Based Financing
Intervention design
• Funds allocation CZK 250 million (app. EUR 10 million)
• Timing December 2012 (call for proposals) → June 2015 (projects
completed)
• Basic features Support creation of employer-based childcare facilities (non-
occasional and non-short-term care) – „Kindergruppen“ for children from 6 months of age.
Employer or a consortium of several employers can apply, services must be provided for employees only.
Kindergruppe size 7 – 24 places, when for size 13-24 some additional (mostly hygienic requirements) apply.
…to be continued
Now, time for your Group Excercise
Group Excercise - Task
• Form groups of 3 – 5 people• Expected result from each group:
7 min presentation, that will explain your solution of the call, including. You have 30 min to prepare.
• 1. Provide definitions of used output indicators on which payments are to be based
• Clearly definable both in terms of quantity and quality• Easy to verify and measure• Close to the desired result as is feasible (significant influence of the
project promoteur, but includes „demand side“)
• 2. How would you collect and verify the data on output indicators?• 3. Do you see any risks linked to your indicators? (Perverse
incentives, possibilities for cheating, gaming, parking, creaming?) How you could mitigate them?
Output Based FinancingCase study, part II:Our solution
Output definition
• Long iterative process, discussion both internal and external
• Using theory of change as a tool• We started from complicated scheme and
reduced the number of output types later from initial six to just two.
Linking Finance to Performance with Output Based Financing
Activities
Activities Participation
Behavioral change
Kindergarten places created
NR of places in kindergartensMax 1000 places10k CZK per place
10 M CZK
Organizations involved in project to create child-care services for
employees
NR of organizationsMax 40 organizations
100k CZK per org4 M CZK
Places available
NR of full-time child-places-monthsMax 1000 children 24 months1500 CZK per child-place per
month36 M CZK
Capacity used
NR of full-time-equivalent child-months in kindergarten
Max 800 children 24 months1825 CZK per child per month
35 M CZK Parents work intesity increases
NR of families where work intensity increased due to kindergaten
Max 500 families15k CZK per family
7,5 M CZK
Kindergartens in operation 6 months after the end of support
NR of places used in kindergartens in operation 6 m after
Max 800 places5 625 CZK per place
4,5 M CZK
Evaluation of the JAP, identification of success factors, best practice identification and disemination...
Evaluation report written, published and circulated1 report3 M CZK
JAP – Child care servicesBudget 100 M CZK = 4 M EUR
Initial thoughts
Kindergarten places created
NR of places in kindergartensMax 1000 places10k CZK per place
10 M CZK
Organizations involved in project to create child-care services for
employees
NR of organizationsMax 40 organizations
100k CZK per org4 M CZK
Places available
NR of full-time child-places-monthsMax 1000 children 24 months1500 CZK per child-place per
month36 M CZK
Capacity used
NR of full-time-equivalent child-months in kindergarten
Max 800 children 24 months1825 CZK per child per month
35 M CZK Parents work intesity increases
NR of families where work intensity increased due to kindergaten
Max 500 families15k CZK per family
7,5 M CZK
Kindergartens in operation 6 months after the end of support
NR of places used in kindergartens in operation 6 m after
Max 800 places5 625 CZK per place
4,5 M CZK
Evaluation of the JAP, identification of success factors, best practice identification and disemination...
Evaluation report written, published and circulated1 report3 M CZK
JAP – Child care servicesBudget 100 M CZK = 4 M EUROutputs
Outputs Outcomes
Outcomes
?
Reduction…
The final model
Services offered to parents
Sustainability of the facility expected
Black box
Higher intensity of labour market participation on labour market in families of parents
Paid unit:Place created.
No payment.
Paid unit:Place occupied. Support decreases in time from 100% in Stage I to 50% in Stage III => road to sustainability after the support ends.
No payment.This is assumption in the design, which is to be tested by evaluation to answer the question if this intervention is worth to repeat.
Preparation of the facility (max. 4 months)
Max 22 months of support
Parents participate, services are provided
Stage I(6 months)
Stage II(6 months)
Stage III(6 months)
Indicator name Places created in kindergruppe
Definition Number of places for kids in quality according to a standard available in newly open kindergruppe. – „Maximum capacity“
Unit Nr of places
Reward per unit 24 114 CZK
Primary data source Monitoring report
Verification source On spot control – observation
Reason Should cover substantial part of investment costs.
Possible perverse incentive or other problems
• There may be tendency to a certain size of kindergruppen – based on production curve or on different requirements for different sizes.
• Quality of kindergruppe „just to meet the standards“
Severity of problems Low
Indicator name Capacity of kindergruppe used
Definition Number of full time equivalent places in each month really attended by client children at least from 70%/40%.
Unit Places x months
Reward per unit 5851 CZK (100% in Stage I, 75% in SII, 50% in SIII, if attendance rate => 70%)(60% in Stage I, 45% in SII, 30% in SIII if => 40%)(0%, if attendance rate below 40%)
Primary data source Monitoring report – online attendance sheet
Verification source On spot control – observation, interviews
Reason Should be the main incentive.
Possible perverse incentive or other problems
Dead souls in attendance sheets. Part time children reported as full time etc.
Severity of problems Moderate
How to set the price• Unit cost were calculated on basis of:
• Unit: Place created• Review of previous similar real-cost based ESF
projects• Market research of compulsory equipment
• Unit: Place occupied• Review of previous similar real-cost based ESF
projects• Occupancy 70% threshold was selected after
investigation of sample of Kindergardens attendance (children on average miss 28 % of time).
• Prices, including methodology how they were calculated, were discussed with Payment and certifying authority (CZ Ministry of finance) and European Commission.
Issues with this method• Project selection
– Criteria of selection are based on objective data (e.g. Share of weighted headcount and capacity of the facility, Regional need, Financial health of the applicant etc.)
– Applicant can calculate number of points awarded in the selection process even before they submit application.
– No external experts are needed for the selection process
• Checks– Costs actually incurred by the facility operator is not taken into
account = no verification of any of the accounting documents – The checks must verify only:
• Quantity of attained units• Compliance with the requirements for quality of the facility (e.g. the
educational requirements for caregivers, eligibility of the target groups, meeting hygiene standards etc.)
• Compliance with the rules for public procurement and the rules for state aid
Lessons learned
Some observations:• Preparation of the intervention scheme took almost a year• Low administrative burden is welcomed both from the side of
applicants and managing authority employees• Only in project selection proces we eliminated cca. 150 man-days of
work.• The interest was lower than expected – only some 30 % of available
funds allocated into 65 projects• Complaints about too low unit prices
• We hope in good sustainability rate as obviously other sources of financing have to be involved.
• Only half of applicants are companies – the rest typically hospitals, universities, research institutes or municipalities
Lessons learned
Some observations:• Tendency to create kindergruppen of 12 or 24 places
• This is response to hygienic requirements. • We were not able to tackle the problem of divergent attitudes of
relevant authorities • This should diminish once the law is in force.
• Unfortunately, more than 20 projects stepped down due to insufficient occupancy and thus lack of financing or other reasons.
• Currently 42 running projects with 770 places. • Based on preliminary evaluation findings, 1 place generates 0,25
FTE of job.• The most important reason for parents to participate is perceived
higher quality of child care in comparison to available alternatives.
Main obstacles for beneficiariesAbsence of legal framework
• There is no specific law for private non-profit child care facilities
• Unclear rules for hygienic and construction standards
Responsibility for the results
• Fluctuating demand from parents for such kind of service– Real demand differs singnificantly from the one declared before the project started –
influence of external factors
• Employers lack sufficient know-how (due to complicated legislation)– They would prefer to use supplier fot the whole project – not allowed
Undervalued units
• It was not meant to be 100% compensation of the real costs– For the purpose of sustainability, financial support declines with every new stage
– Facilities are supposed to be co-financed by beneficiaries or parents themselves
44
Challenges for MA
45
• Monitoring instruments • How to check the occupancy in a simple and reliable way?
• Estimated time for occupancy check with actual instruments is 3-4 hours for one project
• How to work with monitoring system not designed for RBM projects?
• Compliance with the rules for public procurement• How to check in projects based on results and using simplified costs (and
why)?
Ongoing evaluation
46
ESF implementation Department provide ongoing evaluation with the aim:
• To find out, how the projects affect employment of people with small children.
• To find out a motivation of employers to create kindergartens.
• To find out conditions of success and sustainability of the projects.
• To analyze economy of the projects (compared with similar projects based on real costs)
• To analyze the need for childcare services and potential demand for such a kind of projects for 2014-2020
Lessons learnedWhat to do next time better:• Deeper discussion with employers (possible applicants) about the design
and unit prices• Eliminate big steps in payment function as they create danger of perverse
incentives:
0%10%20%30%40%50%60%70%80%90%100%0%10%20%30%40%50%60%70%80%90%100%
Payment dependence on attendance rate
Attendance rate
Pay
men
t re
ceiv
ed
At the level of 70% and 40% even a minimal difference in attendance rate heavily affects the payment receivedMore smooth payment
functions may lower the risk of data manipulation.
Recapitulation1. targeting of subsidy: self-selection of the
employers/applicants
2. accountability: by paying only after delivery of verified outputs = equipment according to standards, only paid when capacity is used to a certain minimal level, risk (demand risk) is shifted to the service provider;
3. innovation and efficiency: “solutions” partly up to service provider.
4. using incentives to serve the intended target group: service offered for lower than market price.
5. output verification and monitoring: on-line attendance sheet, on spot visits,
Recapitulation6. Sustainability:• We expect 4 sources that combined should keep the
facility running Our support in start-up = no need to recover investment
costs Tax deduction established by the new law Employers could provide this service as a highly welcomed
benefit to the employees Parents contribution still expected, but at affordable level.
RecapitulationA. Determining the output Occupied places in the facility
B. Reaching the targer population Selfselection (+ geo targeting, due to fund regulations)
C. Choosing a Subsidy Form Transitional subsidy
D. Determining Value of the Subsidy Ex-ante approximation based on benchmarks
E. Linking Outputs to Subsidy Disbursements
Preparatory phase and Investment costs recovery „place created unit“.
F. Organizing the Institutional Framework Implementing Agendy = Managing AuthorityNo IVA => Verification by MA
G. Evaluating and Mitigating the Project Risks
Not explicitly mentioned
H. Organizing Monitoring and Evaluation On-line attendance sheetsParalel evaluation project
Recapitulation
• OBA could serve well when the problem has a solution in a form of provision of well known and standardizable services.
• „Enhancer strategy“ oriented PMOs should consider use of OBA.
• OBA will not work when the solution is unknown (innovation needed) or unique at the level of individual projects.
• Preparation of the OBA scheme is demanding, but pays-off in simplified administration during implementation.
Linking finance to performance
Some general issues on linking finance directly to performance
Creaming and parking
Other dimensions of performance?
E.g. what does “a kindergarten place” mean?How to define quality?
Take into account need for different services / costs e.g. for job seekers with disabilities versus young recent graduates
Ideally pay for increments rather than thresholds + combine with milestones (=distancetravelled)
Watch out for abuses!
Prefer long contract durations but take measures to pressure for quality and innovation
Beware!
• As Output based financing can shift the motivation of people towards complience with external regulation only (close to amotivation)
• On the other hand, what is a way to transfer funds to a contractor, need not be translated as a way to manage within the contract Leadership within the contractor can still provide an
autonomy supportive environment and may hence get more results than otherwise would be possible
It may be useful to build in some safeguards as to “how” business will be done
57
Further study materials
Excellent overview of OBA in the health sectorhttps://openknowledge.worldbank.org/handle/10986/17194
Global Partnership for Output Based Aid E-learning coursehttps://www.gpoba.org/node/521
• Sourcebook, pp. 153-174• COP RBM OBA Seminar
? Krakow, Poland, September 2014
Further reading
Now it is good time for questions.
60
• Recap on delivery mechanisms
61
How to realise added value?• Enhancers:
Use output based subsidies for basic, stable services ToC to underpin this or to support an input financed approach for large volume
service provision where it merits the investment Keep it simple otherwise (simple planned output-outcome intervention logic) and
support “Toyota Production System”- like service delivery mindset
• Innovators: Use participatory learning approach, structured as an innovation pipeline ToC can be part of developing and testing a concept (e.g. use multiple theories)
• Solutions managers Use outcome mapping, problem driven iterative adaptation, Vanguard redesign,… ToC to get idea of scope / complexity and for evaluation of a particular case (of
what happened, without concluding this will happen again) NOT as a general plan No use for LFA as an approach here either
• static and conflict avoiding approach with tightly specified work programmes that are reported on with “variance” analysis (hiting the targets) not suited for complexity
• but of course stakeholder analysis and problem analysis remain useful as some form of situation analysis is always required
Intervention scopeNarrow Wide
Theoryofchange
Direct/ linear
Complex
Niche resultse.g. basic services(like ambulances,shelters for the poor)
“Transformative” results
Integrated resultse.g. integrated service delivery (like hospital care, education,…)
Ecosystem resultsService delivery combined with more transformative actions enable wider and longer term socio-economic-environmental results.
e.g. changes in societal/organisational norms, policies, practices (tackling local, unique root causes of issues/problems) via advocacy, networks
© Benedict Wauters / Latitude C&T*Adapted from Ebrahim 2010
Intervention scopeNarrow Wide
Theoryofchange
Direct/ linear
Complex
Niche resultse.g. basic services(like ambulances,shelters for the poor)
“Transformative” results
Integrated resultse.g. integrated service delivery (like hospital care, education,…)
Ecosystem resultsService delivery combined with more transformative actions enable wider and longer term socio-economic-environmental results.
e.g. changes in societal/organisational norms, policies, practices (tackling local, unique root causes of issues/problems) via advocacy, networks
© Benedict Wauters / Latitude C&T*Adapted from Ebrahim 2010
Solutions managers drawing on existing services, asking for and introducing
innovative ones
Enhancersmass producing services
Innovation=
complex process of coming up with
service innovations that some day will be
mass produced