Centralized Evaluation and Monitoring of Performance and Risks of Public Investment

Programs

International Seminar on Improving the Quality of Public Investments

and Public-Private Partnerships Brasilia, Brazil

April 25-27, 2005

Background and contents of presentation

Most monitoring and evaluation systems are project focused.

Little evidence of a macro-portfolio approach to M&E of public investments.

Question is how to achieve this?- Is it possible or desirable to create central M&E units for public investment to assess performance from a portfolio perspective.

Could such centralized units be used to monitor fiscal risks inherent in public investment programs. Can government adopt a portfolio approach to managing fiscal risk?

OECD definition of Monitoring and Evaluation Monitoring is a continuous function that uses the systematic collection

of data on specified indicators to provide management and the main stakeholders of an on-going development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds.

Evaluation is the systematic and objective assessment of an on-going or completed project, program, or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process.

Source:OECD, 2002, “Glossary of Key Terms in Evaluation and Results Based Management

Issuers with defining public investment expenditure in a changing budgetary

environment Traditional budgeting systems focused on inputs- separate budgets for

investment and recurrent spending.

Evolution of program and performance based budgeting systems, focused on outputs and outcomes has blurred definitions of public investment.- Increasingly governments encouraged to integrate investment and recurrent budgets in programs.

Also move towards a more medium-term horizon for budgeting MTEF and MTBF increasingly being used to frame annual budget

process Implementation of such systems more successful in some

countries than others. Different countries have different definitions of what constitutes

“investment” versus “recurrent" expenditure.

Other issues affecting M&E of Public investment programs

Donor influences in developing countries Has been a lack of integration in the past. Donors more focused

on their fiduciary duty to own tax payers than to integration of donor spending in beneficiary’s budget system – has led to creation of separate budget monitoring systems in the past

This is changing now with many donors keen to ensure that their funds are integrated in MTBFs still not universal however and problems remain.

Increased use of PPP type investments. off-balance sheet. Increases need for control of these types of investment programs.

Wider range of fiscal risks and performance issues

Main Stakeholders in M&E public

investments Central Planning authorities- Ministries of Finance

and Planning –sometimes separate-increasingly integrated in one ministry.

Line Ministries and spending agencies with

investment oversight responsibility.

Audit and legislative oversight bodies –Parliament- supreme audit institutions etc.

Integrating Monitoring and Evaluation into the budget process

Monitoring and evaluation in performance based budgeting systems Increased emphasis on effectiveness and efficiency

of all expenditures including investment Value for money an important concern Most M&E systems project based

Need systems that measure the overall performance of investment expenditure in meeting overall government objectives.

Question is this possible?

Possible approach- Establish centralized

systems to monitor investment performance Need to assess performance of investment projects in the

context of allocation of scarce resources- Funds should be allocated based on most efficient use of such resources.

Central monitoring and evaluation function can allow for assessment of projects across sectors Allows for projects assessed as poorly performing to be cut

in favor of better performing competing projects with greater likelihood of achieving government priorities.

Establishing centralized units to monitor investment performance assists budget planners and policy makers making allocation decisions during budget planning process. Centralizing aggregate performance of public investment can allow for comparison of performance within and across economic sectors.

Establishment of centralized systems to monitor investment performance

Not suggesting that centralized unit should assess individual project performance. Central units collate the results of monitoring

and evaluation exercises based on standardized criteria- developed by central budget planning authorities.

Supervision and project based monitoring and evaluation best sourced in supervisory agencies and line ministries.

Focus on continuous assessment and regular reporting of overall government investment performance.

Issues related portfolio approach to M&E of Public Investment

Difficulty lies in agreeing common evaluation indicators relevant to different economic sectors. How to compare performance across sectors.-this might be the greatest hurdle to resolve.

May be easier to look at centralized unit to measure fiscal risks than monitor investment performance from a macro perspective.

Perhaps the two functions can be merged in centralized M&E unit

Management of fiscal risks associated with investment portfolios

As with M&E of performance, most discussion of risks of investment programs are project focused.

Little emphasis on monitoring the “@risk” position of government.

“@ Risk” concept derived from value@ risk and cost@risk models originally developed for asset and liability managers in financial markets.

Can models be developed to assess the aggregate @risk position of public budgets inherent in government investment programs?

Value@risk vs Cost@risk models

Both Value and cost@risk models have been developed for financial asset and liability management.

Value@risk models value changes in the capital value (valuation of risk of capital gains and losses).

Cost at risk models developed for sovereign debt management as a way of examining the impact on cashflow –deemed more appropriate by budget planners concerned with annual cash flow rather than capital gains and losses. Particularly appropriate where cash accounting systems are

in place – in many countries Cash is King-risk to capital values of secondary importance

Drawbacks to using @risk models

@risk models offer quantitative measures of risk in a portfolio context.

But Tend to examine maximum exposure based on

normal distribution pattern. Most problems occur during periods of shock –i.e.

outside normal distribution pattern Should only be used as risk management tools –

Still offer budget planners a quantifiable assessment of risks the budget is facing

What are the risks?

Demand risk Construction Risk Policy risk Impact of government guarantees –How to control,

monitor and assess Guarantees major source of fiscal risk but not only one. Increase in expected recurrent costs of investment programs

also source of fiscal risk Moral Hazard issues. E.G. Impact of failing sub-national

government investment programs on national budgets-even without explicit guarantees.

Matrix of potential fiscal risks1

Sources Of Obligations Direct Liabilities Contingent Liabilities

Explicit Government Liability as recognized by Law or contract

Increase in non-discretionary expenditures- In the investment context this would mostly relate to contractual claims where costs of projects are greater than originally envisaged but the government is obligated to fund such increase in costs under the terms of the

contract.

Sovereign Guarantees for non-government borrowing. Particularly important in PPP type investments

Implicit obligation of government reflecting public and interest group pressure

Future recurrent costs of public investment projects

Default of sub-national government or PPP program not explicitly guaranteed by the government but still subject to possible implicit guarantees

1/ Adapted from table 1.1. in Chapter 1 of Government at Risk –Contingent Liabilities and fiscal Risk –World Bank Publication

2002.

Managing investment related fiscal risk limits

in a budget planning context Policy makers should decide on their maximum acceptable

risk threshold Can then allocate fiscal risks across sectors-devolve the

risk To achieve this systems need to be in place to monitor the

evolution of risks over the programming period. Individual agencies responsible for monitoring project risk

and report to central units based on central guidelines. @risk model could offer extra tool in the policy debate

during budget planning process-V@R/C@R limits extra variable to consider

Monitoring and calculation of @risk position -Another argument in favor of centralized

M&E units

Centralized units could collate details of fiscal risks and performance of public investment programs

Units could develop standard risk models to manage such risks

Sectoral line ministries and spending agencies should monitor individual projects and forward details to central agency for collation and reporting on risks and performance associated with investment portfolio

Guidelines on reporting templates and risk management systems should be developed centrally to ensure consistency of approach

Some issues to consider

Should performance and fiscal risks unit be independent of other arms of the executive? Arguments for: Ensures that objectives of government are being monitored rather than objectives of

particular body such a line ministry of MOF. More focused approach to monitoring of performance and fiscal risks than would be

expected of a MOF or MOP with many other priorities-much the same argument as used for establishment of independent debt management offices in the 1990’s.

Arguments against: Another layer of bureaucracy that complicate decision making during budget

planning process Ministries of Finance are best placed to liaise with spending agencies to ensure

receipt of information. Many ministries of finance already have some expertise in performance monitoring

and managing risk

Ultimately dependent on individual country circumstances –As with independent debt offices

Some issues to consider

@risk models still not tuned towards investment programs.

Considerable effort still needs to put into adapting such models for use in monitoring investment portfolio risks.

Standardization of risk management systems across sectors will provide many challenges.

Worth the effort if it assists in the better management of fiscal risks inherent in investment programs.

Thank You