Independent External Evaluation · 2010-03-30 · Evaluation Results and Recommendations The evaluation results and recommendations in this study are based on the following sources

JapanAdministered Account for

Selected IMF Activities (JSA)

I n t e r n a t i o n a l M o n e t a r y F u n d

Independent External Evaluation

February 2010

G L B A LP A R T N E R S H I P SShared Objectives Joint Action Real Impact

International Monetary Fund (IMF)

Office of Technical Assistance Management (OTM)

Independent Evaluation of the Japan Administered

Account for Selected IMF Activities (JSA)

February, 2010

(MURC)

i

Table of Contents

List of Abbreviations ..............................................................................................................iii

Executive Summary Evaluation Results and Recommendations .......................................... vI. Evaluation Results…………………………………………………………………….……...v II. Recommendations…………………………..………………………………...……..…….xiii

Chapter 1 Outline of the Independent Evaluation ......................................................... 11.1 Background and Objectives .................................................................................. 11.2 Implementation and Study Activities ..................................................................... 11.3 Documents, Data and References ........................................................................ 31.4 Evaluation Criteria and Verification Method .......................................................... 51.5 Review of Evaluation Reports .............................................................................. 7

Chapter 2 Outline of JSA and Japanese ODA ............................................................ 122.1 Outline of JSA .................................................................................................... 122.2 ODA Policy of Japan ........................................................................................... 212.3 Consistency of IMF/JSA TA and Japanese ODA ................................................ 23

Chapter 3 Evaluation Results through Data Analysis and Questionnaires .................. 273.1 Review of Evaluation Reporting System of JSA ................................................. 273.2 Efficiency and Cost Considerations .................................................................... 343.3 Evaluation Results of the Questionnaire Surveys ............................................... 40

Chapter 4 Evaluation Results through Field Survey .................................................... 594.1 Evaluation Results of Field Interviews ................................................................ 594.2 Cases of Good Practices .................................................................................... 79

Chapter 5 Overall Evaluation Results and Proposed Rating System .......................... 955.1 Overall Evaluation Results ................................................................................. 955.2 Proposed Rating System .................................................................................. 102

Chapter 6 Recommendations to IMF and MOF ........................................................ 1096.1 Recommendation to IMF .................................................................................. 1096.2 Recommendations to MOF ............................................................................... 111

ii

Appendix

A. Questionnaire Survey Sheet: Beneficiaries B. Questionnaire Survey Sheet: Experts C. Results of Questionnaire Survey: Beneficiaries D. Results of Questionnaire Survey: Experts E. JSA Evaluation Questionnaire (Attachment IV of JSA Guidelines) F. Results of Collected JSA Evaluation Questionnaire G. Schedule of IMF Headquarter meetings H. Schedule of the Field Survey conducted I. List of JSA-funded TAs in the evaluation period

iii

List of Abbreviations

ADB -- Asian Development Bank AfDB -- African Development Bank AFRITAC -- Africa Regional Technical Assistance Center AML -- Anti-Money Laundering APD -- Asia Pacific Department BI -- Bank Indonesia BOT -- Bank of Tanzania CFT -- Combating the Financing of Terrorism DFID -- Department for International Development (UK) EBRD -- European Bank for Reconstruction and Development FAD -- Fiscal Affairs Department FDI -- Foreign Direct Investment FY -- Financial Year GBS -- General Budget Support HQ -- Headquarters IDB -- Inter-American Development Bank IEO -- Independent Evaluation Office IFC -- International Finance Corporation INS -- IMF Institute IT -- Information Technology JICA -- Japan International Cooperation Agency JSA -- Japan Administered Account for Selected IMF Activities LEG -- Legal Department MCM -- Monetary and Capital Market Department MOF -- Ministry of Finance MOU -- Memorandum of Understanding MURC -- Mitsubishi UFJ Research & Consulting NBC -- National Bank of Cambodia NIS -- National Institute of Statistics OAP -- Regional Office for Asia and Pacific OED -- Office of Executive Directors OTM -- Office of Technical Assistance Management PFM -- Public Financial Management PHRD -- Policy and Human Resources Development PRGF -- Poverty Reduction and Growth Facility RA -- Resident Advisor RAP -- Resource Allocation Plan ROSC -- Report on the Observance of Standards and Codes RSN -- Regional Strategy Note SBV -- State Bank of Vietnam

iv

SIDA -- Swedish International Development Cooperation Agency STA -- Statistics Department SWAP -- Sector Wide Approach TA -- Technical Assistance TAIMS -- Technical Assistance Information Management System TCAP -- Technical Cooperation Assistance Program TOR -- Terms of Reference UN -- United Nations UNDP -- United Nations Development Programme UNFPA -- United Nations Population Fund WB -- World Bank WTO -- World Trade Organization

v

Executive Summary Evaluation Results and Recommendations

The evaluation results and recommendations in this study are based on the following sources of information and the analyses conducted by the study team.

The project-level evaluation results from the current IMF/JSA reporting system, i.e. “Project Assessment Report” and “JSA Evaluation Questionnaire”1.

The Questionnaire Survey conducted under this evaluation, addressed to the beneficiaries (recipient governments) of JSA and experts employed by IMF for JSA TA projects.

The results of a field survey conducted by the study team for four countries, Cambodia, Indonesia, Vietnam and Tanzania (East AFRITAC), in August 2009.

The interviews conducted at IMF headquarters mainly with OTM, TA departments and regional departments and OED Japan, in July 2009.

The interviews in Japan with Japanese MOF, OAP and ex-advisors to central banks in Asia under IMF and JICA TA.

The desk analyses of the relevant information collected during this study.

I. Evaluation Results

1. For this evaluation study, the evaluation of IMF/JSA TA for “effectiveness and benefit to the recipient countries” and “accountability to the Japanese general public” are categorized by the terms of reference (TOR) with six criteria: 1) Quality of Activities; 2) Effectiveness and Sustainability; 3) Efficiency; 4) Consistency with Japanese ODA Policy; 5) Visibility of Japan as Donor; and 6) Advantage of IMF Scheme. On the other hand there are six criteria by “OECD DAC Criteria for Evaluating Development Assistance” (relevance, effectiveness, efficiency, impact and sustainability), widely applied to development aid evaluation, have been modified for this evaluation, in seven criteria, namely 1) Relevance, 2) Effectiveness (Impact), 3) Efficiency, 4) Sustainability, 5) Transparency, 6) Consistency with Japanese ODA and 7) Visibility of Japan as a donor. These sets of criteria were applied to conduct this evaluation work.

1. Evaluation Results based on OECD Evaluation Criteria

1) Relevance

2. The Questionnaire Survey conducted by the study team, for both the recipient countries and experts, clearly indicated that they recognize IMF/JSA TA projects as being relevant to recipient government needs and priorities. The answers for consistency with 1 One-page questionnaires collected by OTM within five weeks of the completion of each JSA project, as defined by Guidelines for the Use of Resources for Technical Assistance Activities from the Administered Account for Selected Fund Activities-Japan (JSA Guidelines).

vi

the recipient government policy were either “Excellent” (64%) or “Good” (36%), and there was no answer for “Modest” or “Poor”, as shown below.

Answers by beneficiaries for consistency with government policyAnswer Response Share

Excellent 25 64.1% Good 14 35.9%

Modest 0 0 Poor 0 0

No knowledge 0 0

3. The field survey results indicated that IMF/JSA TA projects are relevant to the recipient countries’ developing priorities, based on their country contexts, and also relevant to IMF priorities, whose regional strategy is indicated in RSNs.

4. According to the interviews with TA departments at IMF headquarters, IMF TA projects are highly specialized and quite focused on the relevant development issues in monetary and finance policies, legislation process and procedures, which make the TA highly relevant compared to TA by other donors.

2) Effectiveness (Impacts)

5. Overall results of the evaluation surveys indicate that IMF/JSA TA projects are basically effective and achieved the desired goals. The results of the “Project Assessment” by TA departments indicate that TA projects have been regarded as effective in light of the targeted objectives. The results of the “JSA Evaluation Questionnaire” (by the beneficiaries) indicated an 80% “yes” response regarding the usefulness of the TA, as shown below.

Answers by beneficiaries for usefulnessAnswer Response Share

Yes 25 80.6% Partially Yes 6 19.4%

6. Based on our questionnaire surveys for the beneficiary countries and experts, the IMF/JSA TA projects have mostly achieved their expected results, with most answers as “excellent” (39%) or “good” (62%), and with no answers for Modest or Poor. TA outputs have been used by authorities and contributed to improvement of knowledge, awareness and motivation for reforms. In addition, TA outputs, to some extent, have directly led to improvement of policies and their instruments in the recipient countries. Major reasons for their effectiveness are: 1) expertise of advisors, 2) experience of experts in other countries, 3) communication skills of experts in understanding country context, and 4) backstopping by headquarters.

7. The field survey results show that IMF/JSA TAs are quite effective in generating (or contributing towards generation) of tangible results. Main reasons for the effectiveness

vii

are: 1) expertise of experts, 2) focus on competitive sectors, 3) selection method (roster system) of experts, 4) communication skills of experts, and 5) backstopping by IMF headquarters.

3) Efficiency

8. Based on our field survey results, IMF/JSA TAs seem to be efficient due to their well-focused approach and selection of competent experts. Overall evaluation results of the study are basically positive in regard to the efficiency of IMF/JSA TA projects.

9. Regarding the efficiency in terms of cost, JSA is well managed through cost-effective means such as the roster-based selection of experts, smooth implementation supported by strong backstopping from headquarters, and repeated assignments by the same expert whose ability is proven. The budget per project and standard cost of long-term experts indicate that these are well within the reasonable range, compared with TA provided by Japanese ODA agencies and other international organizations.

10. The questionnaire survey results for JSA experts indicate that in comparison with other international organizations, IMF/JSA TA is higher in quality by 92%, while the cost is higher by 46%, indicating comparative higher efficiency in terms of quality and cost consideration, as shown below.

Answers by Experts for Comparison with Other International OrganizationsPoints of Difference Answered as “yes”

Quality is higher for IMF/JSA 92% Monthly remuneration is higher for IMF/JSA 46% 4) Sustainability

11. According to our questionnaire surveys for both recipient countries and JSA experts, major factors for sustainability of TA projects such as institutional memory, staff capabilities, or budget allocation are achieved in most of the counterpart authorities, as shown below. The majority of the experts have expressed their shared views that sustainability of IMF TA depends on a willingness to reform by authorities.

Answers by Experts for Levels of SustainabilityAnswer Response Share


Modest 11 13.4% Poor 1 1.2%

No knowledge 7 8.5%

12. The field survey results indicate that the sense of ownership by recipients varies by country, e.g. Vietnam with strong ownership. The concept of increasing the sense of

viii

ownership for countries with weak ownership needs to be addressed to support the sustainability of the TA contribution.

13. As for financial sustainability, it is forecasted that IMF needs to rely more on externally financed sources, due to decreasing internally funded sources. IMF has introduced a charging system for internally-funded TA, which might not be significant in terms of volume, but it will have an impact in enhancing ownership of the recipient countries. Therefore, an application of the charging system to JSA will be needed to be considered.

5) Transparency

14. Although the IMF/JSA TA projects deal with policy-level and highly specialized areas of finance and monetary systems, the field survey results indicate that there is room for improvement in sharing information with other donors, including Japan, regarding the activities of IMF/JSA TA projects.

15. The recruitment system to select experts for IMF TA is by a roster system, due to the limited number of projects and competent experts to be employed. Although the current system is considered to be reasonable for operational efficiency, enhancement in transparency of the recruitment system is needed.

6) Consistency with Japanese ODA

16. Regarding the consistency of JSA with the Japanese ODA policy, three main documents compiled by the Japanese Government (ODA Charter, Mid-term Policy and Country Assistance Programs) were examined. The result of this examination was an assessment that IMF/JSA TA is highly consistent with the Japanese ODA policy.

7) Visibility of Japan as a donor

17. The overall evaluation results indicate that the visibility of Japan as a donor for IMF/JSA TAs is relatively low compared with the high scores gained for other aspects of evaluation and for effectiveness and benefit to the recipients.

18. The Questionnaire Survey results show that Japanese sponsorship of JSA TAs is known by 71% of the high ranking officials in the recipient governments, recognizing JSA is financed by Japan. On the other hand, 96% of experts answered that they are aware of the Japanese sponsorship. The narrative comments by experts indicated their support and understanding on the need for enhancing visibility of Japan as a donor.

19. The results of the field survey revealed that in particular countries, e.g. Cambodia, visibility is very high on the recipient side as well, partly because of the volume of JSA TA delivered and the IMF-Japan (MOF) joint mission’s visit in early 2009. However, in other countries visited, e.g. Indonesia, Tanzania and Vietnam, the awareness on the recipient authorities was low, suggesting some room for improvement.

ix

Answers by beneficiaries for recognition of Japan as a donorAnswer Response Share

Yes, very much so 12 30.8% Yes, relatively so 13 33.3%

Not so much 5 12.8% Not at all 1 2.6%

No knowledge 8 20.5%

2. Evaluation Results Based on TOR criteria

1) Overarching Objectives

20. The overarching objectives are defined by the three points below. (1) To assess whether the JSA has been an effective instrument for capacity building in

countries, covering issues of (a) quality of TA, (b) sustainability, and (c) efficiency; (2) To determine whether the administration of and information on the JSA is sufficient to

provide accountability and whether the JSA provides good value for money, covering issues of (a) consistency with Japanese ODA, (b) visibility of Japan as a donor, (c) supporting reasons for IMF TA; and

(3) To make recommendations to improve the JSA in furtherance of the objectives in (1) and(2) above.

2) The JSA as an Effective Instrument for Capacity Building in Countries

21. The followings are findings by the evaluation team for each of the subjects being questioned.

(a) Quality of JSA TA 22. Is the JSA TA of high technical quality consistent with international best practice and member country priorities?

Most of the recipient country counterparts recognize that the quality of IMF/JSA TAs is at the best international level.

Most of the recipient countries also agree that IMF/JSA TA's are quite consistent with their developing priorities.

23. Is the involvement of area departments useful in determining member country TA priorities, integrating them with Fund surveillance and lending activities, and providing quality control?

The involvement of area departments is useful and critical because initial identification of needs is done by the area departments. They monitor surveillance results and consider what kind of TA activities and areas are necessary. Area departments compile Regional Strategy Notes (RSN) which outlines regional issues to be addressed and assistance priorities and plans in each country.

x

24. What measures are in place in TA departments to ensure that IMF/JSA TA is timely and of high quality? Have these measures been adequate/effective?

TA departments provide a strong “back stopper” service, which is key for quality control of JSA and other TA activities. The headquarters staff in TA departments are specialists themselves, who have deep knowledge and experience in the subject areas. This backstopping function differentiates IMF TA from other donors, where usually, the generalist staff manages TA activities. As for timeliness of TA, JSA is a quick funding source compared with other externally funded TAs.

25. Are the current evaluations/assessments of JSA activities adequate/effective? The current evaluation/assessment system of the IMF/JSA TA seems to have much

room for improvement from the viewpoints of both “accountability” and “management improvement.”

(b) Sustainability of Results Achieved Through IMF/JSA TA 26. Have JSA activities as stand-alone, or in conjunction with IMF financed activities, achieved tangible and sustainable results in beneficiary countries?

Most recipient country counterparts recognize that the implemented IMF/JSA TAs have been able to generate tangible outcomes, in some cases impacts, in their countries.

The “coverage” of IMF/JSA TA, however, varies with countries quite widely.

(c) Efficiency and Cost Considerations 27. Is IMF/JSA TA a cost-effective way to deliver capacity building assistance to countries, especially in relation to other comparable TA delivery modes (as determined by the evaluators), and bearing in mind the difficulties inherent in measuring the benefit of capacity building activities?

Since many (most) recipient countries regard the contents of IMF/JSA TA as unique, the comparison with other developing partners seem not to be relevant.

However, if we dare make some comparison, there seem to be several factors which contribute to the cost-efficiency of JSA. First, the roster-based selection process of experts is more cost-efficient than public bid or open application procedures. Second, the repeated assignment of the same qualified expert whose ability is proven, is efficient in many respects due to the established relationship and experience in the field. Third, strong support and quality control of IMF headquarters’ functional departments contribute to the smooth and efficient implementation of IMF/JSA TA. The standard cost of dispatching the expert, as far as the remuneration rate is concerned, seems to be reasonable compared with similar Japanese TA schemes.

28. Is the cost of IMF/JSA TA reasonable in relation to other comparable TA providers? The Budget per project (around US$150,000) and the standard monthly remuneration

rate for dispatching experts as determined in the JSA Guidelines seems to be reasonable compared with similar cases in Japanese bilateral technical assistance such as JICA. Furthermore, the disbursement rate is quite high, 96% on average from 1993 to 2009, which is another aspect of sound management and administration by the IMF department in charge.

xi

3) IMF/JSA TA Delivery is Providing Good Value for Money for Japanese Taxpayers

(a) Consistency of JSA Activities with Japan’s ODA policy 29. Have JSA activities been consistent with Japan’s ODA policy?

There are three major documents compiled by the Japanese Government which represent the Japanese ODA policy, i.e. ODA Charter, Mid-term Policy and Country Assistance Programs. Examination of these documents show that IMF/JSA TA activities are highly consistent with the Japanese ODA policy.

30. Have the JSA activities overlapped negatively with other Japanese ODA initiatives, e.g., bilateral ODA and/or activities of “Japan funds” at other international institutions? If so, how can these overlaps be best removed/addressed?

None of our findings indicate the existence of duplication or overlap between the IMF/JSA TA and other Japanese ODA technical assistance implemented. This is partly due to the difference in areas of specialty focused on by the IMF and Japanese ODA. Nevertheless, there is an important and necessary complementary role for IMF/JSA TA in light of future development of Japanese bilateral ODA in the financial and fiscal sectors.

(b) Visibility of Japan as a Donor 31. Has the JSA provided adequate visibility to Japan as an ODA donor country?

Partly yes and partly no. There is room for improvement, in particular on the beneficiary side, with more institutionalized arrangements by the IMF and the Japanese authority in charge.

32. Has the JSA enhanced the visibility of Japan as an ODA donor country? Partly yes and partly no. Since Japan is recognized as a large donor, in particular in

Asia, the effectiveness of enhancing visibility through JSA is, if properly implemented, is marginal. However, in countries where the Japanese presence is not strong, e.g. Africa, utilization of JSA, as a part of IMF which has strong influence in the financial and fiscal sector, might have more effectiveness.

33. Have the “publicity” guidelines in the JSA been effective? If not, how can they be changed to improve the visibility of Japan as an ODA donor country?

There seem to be various possibilities for improving Japan’s visibility including publicity activities by building into administrative process. This is discussed under recommendations in the next chapter.

(c) Support for IMF TA Versus other Avenues for Providing Development Assistance 34. What are the advantages of supporting IMF TA, as compared to using Japanese funds for other development assistance?

For Japan, the areas involving financial systems, fiscal management and macroeconomic analysis are areas with limited human resources that need to be supplied in TA projects to developing countries. The advantage of sponsoring IMF TA through JSA is that it expands the opportunity for Japanese bilateral ODA coverage,

xii

with possible collaboration and complementary exchange among experts and ODA agencies; typically JICA. IMF TA in African countries. e.g. Tanzania, is regarded in Japan as a pioneering effort, compared to some countries in Asia, e.g. Vietnam, where the country is already utilizing JICA TA in the central banking and fiscal areas. A strategy can be implemented to make use of JSA for pioneering efforts in a less experienced country. The coverage of bilateral donors will increase when Japanese ODA catches up in providing its own TA directly to the recipient countries. (see the illustration of this concept below)

Complementary Role of IMF TA and Bi-lateral TA

xiii

II. Recommendations

Recommendations to IMF

1. Improvement in Transparency

35. It is recommended that the results of project-level evaluation by recipients, collected through JSA evaluation questionnaires, should be summarized and reported with more detailed analysis of the evaluation results in the JSA Annual Report in order to present this information to the general public both in Japan and in other countries.

36. In order to maintain the current recruitment system which is considered to be operating efficiently, it is recommended that there be disclosure of the recruitment process for experts to be employed by the IMF for JSA TA. Particular emphasis should be placed on the specific qualifications required, relevant background and experience along with communication skills with recipient agencies and IMF headquarters. This will help to increase transparency in the recruitment procedure since the recruitment process does not utilize an open-bid system similar to other donor agencies.

37. It is recommended that information sharing with other donors regarding the contents of TA activities be institutionalized. This will provide enhanced transparency and possible complementary effect with the other donors’ input, in particular with Japan, i.e. the Japan International Cooperation Agency (JICA). Since there might be a limitation in information disclosure with the authorities in the recipient countries, the level and content of information sharing must be considered in light of country and sector context as well as potential interest by other donors in the field. The IMF Resident Representative should assume a coordinating function in the field.

2. Improvement of Management

38. It is recommended that management of JSA Project Proposals and Project Assessments be improved regarding the points listed below:

Improvement in consistency in the information to be filled-in; Improvement in filling out “verifiable indicators” for the projects with guidance regarding

the intention and the concept; Improvement in describing the assumptions for the project; Improvement in filling in the matrix for the mid-term objectives; Improvement in describing the achievement of the projects and their intended targets; Improvement in procedures regarding the extension of projects with conditions to be

clarified.

39. In order to enhance the effectiveness, impact, and efficiency of projects, it is

xiv

recommended that a strategy should be established for programming the TA project in combination with long-term and short-term (peripatetic) experts, depending on the level of focus for a particular issue in a particular country, with attention toward:

More focus regarding both the “IMF Context” and the “Country Context” in selecting the country and areas of TA.

An exit policy and targets that need to be achieved within a limited period. This will enhance ownership of the recipient, which is also indicated in the 2005 IEO Report as a need for graduation of long-term experts.

40. It is recommended that more focus on “sustainability” and “ownership” of the recipient organizations occur by incorporating them in the JSA Proposals as well as in the Project Assessment, through applying methodologies in objective evaluations.

41. It is recommended that OTM improve the managing and administering JSA project proposals, project assessments, and JSA evaluation questionnaires, through a more efficient system to store, file, collect, and sort data on the IMF intranet. While we discovered that OTM started its improved system for collecting the questionnaires, more attention is necessary to sustain the effort given the limited resources in OTM.

3. Improvement of Visibility1) Symbolic Improvement

42. It is recommended that a logo and/or concept be printed in support of the JSA and this logo should appear on IMF/JSA TA relevant materials, documents, reports and name cards of all the experts, to indicate clearly that JSA is being financed by Japan.

2) Enhance Visibility of the Recipients

43. As the results of this evaluation study indicate, the recipient authorities need to be more informed of the Japanese contribution to IMF/JSA TA projects. In this regard, it is recommended that the IMF enhances its procedures, so that the recipients of JSA TA, at various levels, are well-informed of Japan as the donor.

3) Enhance Japan’s Involvement in JSA projects

44. It is recommended that the recruitment of Japanese experts for short-term assignments for seminars and workshops be strengthened, especially for assignments of less than one week. While the current number of Japanese experts in the roster is 30 out of a possible 1,350 in total as of June 2009, cooperation with Japanese authorities is needed to up-date and improve the roster.

45. It is recommended that regional forums are organized and conducted in order to introduce activities covered by IMF/JSA TA in Asia and/or Africa. These forums are to be

xv

held in Japan, with the participation of IMF experts, recipient governments, The Japanese Government, ODA agencies, Japanese experts, scholars, the private sector and the mass-media. This will facilitate the: (i) dissemination of information focused on relevant experience, (ii) publicizing JSA’s financing support, and (iii) strengthening communication with the Japanese ODA circle.

Recommendations to MOF

1. Improvement of Transparency and Information Disclosure

46. It is recommended that public relations should be strengthened in order to relay information on JSA projects and their achievements to the general public, i.e. taxpayers, through seminars, forums, websites, the writing of articles on ODA publications and other media.

47. It is recommended that Information sharing should be enhanced regarding Japanese ODA agencies and relevant groups, academic circles, experts and consultants, with a focus on project-level information for JSA projects. This will lead to the identification of interesting issues to be covered by the proposed regional forum.

48. It is recommended that a strategy is created to make use of IMF TA and the interests of Japanese ODA in the field of financial and fiscal system development in developing countries with attention to:

Policy of MOF and MOFA for ODA policy, utilization of Japanese resources and utilizing IMF capacity in a complementary arrangement.

Strategy to use the JSA as a tool to supplement Japanese ODA. Develop documents to explain the Japanese historical experience in monetary and

fiscal systems, and policy implementation experience as reference material for developing countries.

Establish a strategy to mobilize JSA and JICA experts, depending on the country context in development needs of the target countries’ financial and fiscal system, and possible involvement of Japanese bilateral assistance.

2. Improvement of Visibility

1) Symbolic Improvement

49. It is recommended that cooperation with IMF occur in the symbolic improvement of JSA related documents to be used by IMF staff as well as experts. In order to have the recipient government be aware of the Japanese assistance, the use of bilateral channels to inform the recipient government can be effective.

xvi

2) Enhance Japan’s Involvement in JSA Projects

50. It is recommended that the number of Japanese experts on the IMF roster of experts be increased with screening by capacity and appropriate tasks to be covered. Despite the efforts by IMF and MOF, in the past to increase Japanese participation, use of Japanese nationals for IMF/JSA TA is quite limited. In order to improve the participation of Japanese experts, more participation of officials from MOF, Bank of Japan and Financial Services Agencies should be encouraged with cooperative effort by the respective departments among Japanese agencies in seeking capable staff as IMF/JSA TA experts.

51. In order to increase awareness by Japanese domestic interested parties, it is recommended regional forums be organized to disseminate information on the activities of JSA experts, in particular, with countries that have limited information about the Japanese ODA circle.

52. It is recommended that a system to have JSA experts establish communication with JICA at headquarters as well as field offices should be set up. This could include possible collaboration in the field.

1. Outline of the Independent Evaluation

1

Chapter 1 Outline of the Independent Evaluation

1.1 Background and Objectives

Background53. Since 1990, the Japanese Government has provided financial support for IMF technical assistance to its member countries to strengthen their capacity to formulate, implement, and maintain macroeconomic and structural adjustment programs. Japan’s contributions are provided through the Japan Administered Account for Selected IMF Activities (JSA), and Japan has been the largest contributor to the IMF’s technical assistance (TA) activities

54. As of the end of 2008, Japan has made contributions totaling approximately US$279 million to finance IMF TA activities. These funds have been used to cover costs of selected IMF TA projects, including the salaries and travel costs of short- and long-term TA experts and costs associated with organizing seminars and workshops. 50 percent of each annual JSA contribution is targeted at TA activities for countries in the IMF’s Asia and Pacific departments and selected countries in Central Asia. The use of JSA funds has been flexible, with the main focus of the IMF/JSA TA program being support for low- and lower-middle-income countries in building institutions and capacity to implement growth-enhancing economic policies.

Objectives55. This evaluation is being requested and sponsored by Japan’s Ministry of Finance, in its capacity as the Japanese agency responsible for the JSA, and is the first evaluation of its kind. The evaluation’s overarching objectives are:

(1) To assess whether the JSA has been an effective instrument for capacity building in countries, covering issues of 1) quality of TA, 2) sustainability, and 3) efficiency;

(2) To determine whether the administration of and information on the JSA is sufficient to provide accountability and whether the JSA provides good value for money, covering issues of 1) consistency with Japanese ODA, 2) visibility of Japan as a donor, 3) supporting reasons for IMF TA; and

(3) To make recommendations to improve the JSA in furtherance of the objectives in (1) and (2) above.

1.2 Implementation and Study Activities

56. Mitsubishi UFJ Research & Consulting Co., Ltd. (MURC) was appointed by IMF as a consultant to conduct this independent evaluation through a contractual arrangement between IMF and MURC by Agreement Number 1507, dated June 8, 2009, and MURC, as a contractor, is responsible for setting up study plans, conducting data/ information collection


2

and their analyses, implementing field surveys and questionnaire surveys, and reporting. MURC formed a team of experts listed below to conduct this task augmented by one outside advisor from the Japan Economic Research Institute (JERI). The flow of the work conducted is depicted in Chart 1.1, and brief contents of the work are described below.

Evaluation Team MembersMr. Hidekazu Tanaka, Team Leader (MURC) Mr. Jun Arakawa, Program Evaluation Expert (MURC) Mr. Yasuhiro Sakon, Program Evaluation Expert (MURC) Mr. Yoichi Hara, Financial Management Specialist (MURC) Ms. Masumi Shimamura, Aid Policy Specialist (MURC) Mr. Hisatsugu Furukawa, Advisor (Japan Economic Research Institute)

Chart 1.1 The Evaluation Study Flow Chart

Preparation and Interviews in Japan57. The work started with listing and sorting of JSA projects during the evaluation period with data available from the IMF web-site as well as information provided by OTM. At the same time, review of annual reports, project proposals and assessment reports for each JSA project were conducted, and review of the past evaluation reports for IMF TA was also covered. Interviews with relevant organizations and experts in Tokyo took place with the Ministry of Finance, Regional Office for Asia and the Pacific (OAP), former IMF TA advisor to the National Bank of Kyrgyz Republic and the former JICA TA advisor to the State Bank of Vietnam. These interviews provided the team members with basic knowledge and


3

ideas of JSA and TA in the areas concerned by IMF, in developing countries and transition economies.

Discussion at IMF HQ58. The MURC Evaluation Team, with all five members, visited IMF Headquarters from July 13 to 17, 2009, to have discussions with the OTM, and conducted interviews with key IMF staff at Regional departments, MCM, FAD, STAT, the Legal Department, IEO, IMF Institute and the Executive Director’s Office. The arrangement of interviews and coordination of the team was managed and assisted by the Office of Technical Assistance Management (OTM), by the following contacts. Mr. Alfred Kammer, Director, Office of Technical Assistance Management (OTM) Mr. Harish Mendis, Senior Technical Assistance Officer, OTM Mr. Barrie Williams, Senior Budget Assistant, OTM

Field Visit Survey59. In order to verify the achievement of results, effectiveness, and efficiency of IMF/JSA TA projects, visit interviews were conducted in Indonesia, Cambodia, Vietnam and Tanzania during August 17 to 26, 2009. The evaluation in the field was focused on the benefit and effectiveness to the recipient countries and visibility of Japan as a donor. Through the field visits, good practice cases were selected for each country as the result of the interview in the field.

Questionnaire Survey60. The questionnaire surveys were conducted through e-mail and web-based collection of answers from recipients as well as experts involved in TA activities financed by JSA. The questionnaire distribution, collection and compilation of the results were conducted by Rapid Access International (RAI), a Washington DC based consultant, through a sub-contracting arrangement by MURC.

1.3 Documents, Data and References

61. The Team reviewed the documents related to activities of JSA and evaluation work done by IMF and the World Bank, as listed below.

1) JSA Annual reports (2004-2008): available through the IMF website

English version: http://www.imf.org/external/pubs/ft/ta/index.asp Japanese version: http://www.imf.org/external/pubs/ft/ta/jpn/index.htm

Annex 1 of each Annual Report 2004 to 2008 was reproduced in a form using Excel, by the team, to be used for further analysis.


4

2) JSA Project Proposals and Assessments: files were provided by OTM

FY 2004 (May 2003-April 2004) FY 2005 (May 2004-April 2005) FY 2006 (May 2005-April 2006) FY 2007 (May 2006-April 2007) FY 2008 (May 2007-April 2008)

Note: The assessment reports are not available for the projects being extended to the present or for projects cancelled.

3) Beneficiary Survey Results 62. For each JSA project, at the time of project completion, a “Beneficiary Survey” sheet is distributed for the beneficiaries (counterpart recipient organization) to respond with checks and narratives or notes. The team obtained the returned sheet copies through OTM.

4) Reference Project Documents 63. The JSA project documents, back-to–office reports, mission report, presentation material and related documents for selected JSA projects were made available from TA departments through OTM to the evaluation team.

5) Evaluation Reports 64. Among several evaluation reports for IMF TA, the following were considered as highly relevant to the work, and carefully reviewed by the team. The review results for some of these reports are provided in Chapter 1.5.

“Enhancing the Impact of Fund Technical Assistance”, April 3, 2008, Office of Technical Assistance Management (In consultation with other Departments), IMF

“Evaluation of the Technical Assistance Provided by the International Monetary Fund”, January 2005, Independent Evaluation Office, IMF

“Evaluation of the Technical Assistance Subaccount for Iraq”, March 2008, Bruce Murray (Team Leader)

“Independent Evaluation: African Regional Technical Assistance Center (AFRITAC), March 2009, Bruce Murray (Team Leader)

65. The report below is an evaluation for the Japanese trust fund for TA activities under PHRD, a reference for a similar contribution to the World Bank.

“Evaluation of the Policy and Human Resources Development Trust Fund”, January 2008, UNIVERSALIA


5

1.4 Evaluation Criteria and Verification Method

66. The evaluation work was conducted through: 1) Review of IMF/JSA TA activities; 2) Evaluation of JSA TA projects; and 3) Field visit interviews to recipient countries to determine the overall scope of IMF/JSA TA activities. The following methodology was applied for each of these tasks.

1) Review of IMF/JSA TA Activities 67. The holistic review of the IMF/JSA TA activities was conducted from a multi-dimensional viewpoint. Based on the contents reported in the Annual Report, the general characteristics of the past JSA TA activities were analyzed by areas, regions/countries, and year. In particular, among the three main areas listed below, the subjects covered, as well as the activities of countries with repeated TA recipients.

Chart 1.2 Core Areas of IMF Technical AssistanceAreas Subjects Fiscal Policy and Management Tax policy, Tax and customs administration, Expenditure policy,

Budgeting and public expenditure management, Fiscal management, Fiscal federalism

Macroeconomic and Financial Statistics

Multi-sector statistical issues, Balance of payments and external debt statistics, government finance statistics, Monetary and financial statistics, National accounts and price statistics, Data dissemination standards

Monetary Policy and Financial System

Central banking and currency arrangements, Monetary and exchange policy operations and public debt management, Financial market development, focusing particularly on money, government debt, and foreign exchange markets, Exchange systems and currency convertibility, Payment systems, Bank supervision and regulation, Bank restructuring and banking safety nets

Other Areas Legal, Training and Anti-Money Laundering/ Combating the Financing of Terrorism

2) Evaluation Criteria of IMF/JSA TA Projects 68. For this evaluation study, the evaluation of IMF/JSA TA for “effectiveness and benefit to the recipient countries” and “accountability to the Japanese general public” are categorized by the terms of reference (TOR) with six criteria: 1) Quality of Activities; 2) Effectiveness and Sustainability; 3) Efficiency; 4) Consistency with Japanese ODA Policy; 5) Visibility of Japan as Donor; and 6) Advantage of IMF Scheme. On the other hand there are six criteria by “OECD DAC Criteria for Evaluating Development Assistance” (relevance, effectiveness, efficiency, impact and sustainability), widely applied to development aid evaluation, have been modified for this evaluation, in seven criteria, namely 1) Relevance, 2) Effectiveness (Impact), 3) Efficiency, 4) Sustainability, 5) Transparency, 6) Consistency with Japanese ODA and 7) Visibility of Japan as a donor. These sets of criteria were applied to conduct this evaluation work.


6

Chart 1.3 Evaluation Criteria of IMF/JSA TA Projects Category Viewpoint

Effectiveness and benefit to the recipient countries

Quality of Activities(1) Quality and consistency with the priority of recipients (2) Effectiveness of regional departments (3) Quality and timelines of Technical Assistance

Departments (4) Evaluation to JSA TA projects for relevancy and

effectiveness Effectiveness and sustainability(1) Achievement of issues through TA (benefit to the recipients) Efficiency(1) Efficiency (value for money) of JSA TA projects (2) Appropriateness of cost for JSA TA projects

Accountability to the Japanese general public

Consistency with ODA Policy(1) Consistency with Japanese ODA Policy (2) Duplication with other technical assistance by Japan,

including measures to be taken Visibility(1) Level of visibility of JSA TA projects (2) Improvement of visibility through JSA TA projects (3) Effectiveness of guidelines for improving visibility Advantage of IMF scheme(1) Advantage of IMF scheme compared with other schemes

69. In regard of the effectiveness, the following “logic model” is taken into account as a base:

Chart 1.4 Basic “Logic Model” for IMF/JSA TA Projects

External factors

Inputs Activities Outputs Outcomes Impacts

JSA Funds:

IMF Experts:

HQ Resources:

TA Activities & Outputs:

- Capacity building (Human, Institutional)

- Materials, seminars, WSs, daily guidance

- Policy designing

- Laws, regulations, manuals

- Management & organization reforms

- System architectures

Immediate Intermediate Final

TA Outcomes:

- Policy-making & -reforming capacities

- Human & Institutional

- Including management & system

-related capacities

- Policies and/or policy reforms in place

- Laws, regulations, manuals

- Management & organization reforms

- System architectures

TA Impacts:

- Socio-economic

changes and/or

improvements

Major targets of this evaluation


7

70. The questionnaire survey was conducted and addressed to counterparts of recipient governments as well as experts sent through JSA projects. The questionnaires, covering the above points, were sent via e-mail instructing respondents to submit responses to the web-site created for this evaluation work. The contents of the questionnaires were coordinated through consultation with OTM and e-mail addresses were provided by relevant TA departments of IMF.

3) Field Visit Interviews to Recipient Countries 71. Taking into account the above evaluation coverage, the selection of the countries to be visited was based on the usage of IMF/JSA TA, intensity of Japanese bilateral ODA, and availability of the counterparts during the designated visit period. For each field visit, conducted by two team members to each country, interviews were made based on criteria to the recipient ministries and agencies, IMF resident representatives, technical assistance center, experts under JSA and other donors including Japan and the World Bank.

1.5 Review of Evaluation Reports

72. As a reference to the evaluation work, some of the past evaluation reports closely related to the evaluation of JSA were reviewed and their brief contents are introduced in this section. The reports reviewed were 1) Evaluation of IMF TA by IEO, 2) Enhancing the Impact of Fund TA by OTM, and 3) Evaluation of the PHRD Fund of the World Bank.

1) “Evaluation of The Technical Assistance provided by The International Monetary Fund” prepared by the Independent Evaluation Office of International Monetary Fund, January 31, 2005

73. In this evaluation work, the Independent Evaluation Office (IEO) of IMF assessed its own technical assistance (TA) to its member countries. The methodologies used were desk reviews of relevant data as well as in-depth case studies for six selected countries; Cambodia, Honduras, Niger, Ukraine, Yemen, and Zambia. The case studies for six countries were compiled based on the field visits and interviews in each country with public officials and donors, reviews of past evaluations, and interviews with IMF staff and other stakeholders. In the evaluation analysis, the TA process was examined by dividing it into three stages; (1) Prioritization: How are countries' TA needs identified? What can be done to make the process more strategic so as to increase the relevance of IMF TA activities? (2) The delivery process: What factors influence the effectiveness of the various modalities for TA delivery? (3) Monitoring progress and evaluating impact: How is progress tracked and what factors contribute to the impact of TA?

74. Major findings and recommendations are summarized as follows: Prioritization: Seventy percent of IMF TA is provided to countries with per capita income


8

below $1,000, and therefore, IMF TA is well targeted to low-income countries. Furthermore, the report analyzed the level of linkage between TA priorities and key policy issues identified in Article IV consultations. Regarding the linkage with Article IV consultations, the evaluation report presents interesting analysis in its Chapter 4 (Table 4.1. on page 45), where the strength of the link between Article IV consultations and IMF TA is examined for 18 countries. Chart 1.5 provides a summary of the scores.

Chart 1.5 Strength of the Link between Article IV Consultations and IMF TA

PEM andBudget

Revenue (Tax and Customs)

Monetary and Exchange

Rate Policy

Banking Supervision

and Restructuring

Statistics Average Score

Bangladesh 1 2 3 1 2 1.8 Belarus 2 2 1 2 1 1.6 Bulgaria 2 3 3 2 2 2.4

Burkina Faso 3 1 n.a. n.a. 1 1.7 Cambodia 2 3 3 3 3 2.8 Cameroon 3 3 n.a. n.a. 1 2.3 Ecuador 2 2 1 3 2 2.0 Ethiopia 1 3 2 3 3 2.4

Honduras 1 3 1 3 2 2.0 Jamaica 3 1 1 3 2 2.0 Morocco 3 3 2 2 2 2.4

Niger 3 3 n.a. n.a. 1 2.3 Tonga 2 1 3 3 2 2.2 Turkey 2 3 3 2 2 2.4 Ukraine 3 3 3 3 3 3.0 Vietnam 1 1 2 3 2 1.8 Yemen 3 3 2 3 2 2.6 Zambia 3 1 3 3 3 2.6

Average score 2.2 2.3 2.2 2.6 2.0 2.3 Source: IMF documents. 1 = weak or no apparent link; 2 = partial link; 3 = close link between surveillance and TA from IMF; n.a. = not applicable (for these countries, responsibility for subject area rests with the regional central bank (Banque Centrale des Etats d’Afrique de l’Oeust).

(Source: Evaluation of The Technical Assistance provided by The International Monetary Fund, IEO-IMF, 2005, p.45)

75. On average, the strength of the links vary from country to country with a range of 1.7 at the lowest end to 3.0 at the highest and total average of 2.3. (The designation of this 3-point scale is as follows: 1 = weak or no apparent link; 2 = partial link; and 3 = close link between surveillance and TA from IMF.) It should be noted that Cambodia, where JSA plays a significant role and is the subject of our field survey this time, obtained an average of 2.8, which is the second highest after Ukraine. Cambodia reached the full score of 3.0 for the four TA areas (Revenue, Monetary and Exchange Rate Policy, Banking Supervision and Statistics) and 2.0 for PEM and Budget. This result implies that, given the magnitude of JSA in the country, IMF TA is very appropriately designed and provided with the right priority-setting in Cambodia. On the contrary, it is noted that the score for Vietnam is rather low with a 1.8 average. However, the strength of the link differs sharply according to TA areas, i.e. there is a close link in banking supervision, but partial or weak links in other


9

areas.

76. The analysis of short-term versus long-term modalities for providing TA is another interesting issue noted in the report.2 This raises a question regarding the result of the 1999 Fund-wide evaluation which suggested that shorter term modalities were more cost effective. The report argues that long-term experts (LTE's) can play a critical role and provide significant values to recipient countries. As evidence of this line of argument, the report presents the cases in Niger, Cambodia, Honduras and Ukraine, where the works of LTE's were highly appreciated by the counterpart officials. It concluded that there is an appropriate mix between resident experts and short-term experts depending upon the level of institutional capacity and country context.

77. The delivery process: The recipient countries have generally been satisfied with the experts sent by the IMF. They appreciate the experts’ hands-on role in training and coaching, accessibility, and emphasis on teamwork. The report recommends enhanced involvement of the authorities in the preparation of terms of reference (TOR), which led to stronger ownership and shared expectations by the authorities and IMF about final objectives.

78. Monitoring progress and evaluating impact: Progress has generally been achieved in enhancing the technical capacity of the agencies that the IMF typically supports. The report recommends that IMF documentation and reporting be improved to track the different stages of progress toward meeting the final objectives of TA, with clear indicators of progress at the outset. Also, the report suggests a need for unbundling between short- and medium-term indicators that capture different stages of the results chain, which provides a track record in implementing TA in making decisions about future TA.

79. In conclusion, key recommendations were made as follows: The IMF should develop a medium-term country policy framework for setting TA

priorities, incorporating country-specific strategic directions and linked to more systematic assessments of factors underlying past performance.

The IMF needs to engage countries to help them articulate their medium-term capacity-building needs in the IMF's areas of responsibility and in accordance with the PRSP.

The IMF should develop more systematic approaches to track progress on major TA activities and identify reasons behind major shortfalls.

Greater involvement by the authorities and counterparts in the design of TA activities and arrangements for follow-up should be emphasized as a signal of ownership and commitment.

2 This analysis is presented in detail in “B. Short-Term Versus Long-Term Modalities for Providing TA” (pp.73-75) and in “C. Attributes of a Good Long-Term Expert” (pp.76-77) of the IEO Report.


10

80. In addition to the major report, there are case studies on specific countries followed by recommendations. For example in Cambodia, some of the recommendations are as follows:

In order to avoid countries becoming “addicted” to TA it is necessary to plan how to phase out TA advice in order to encourage the authorities to start walking on their own, and to help them “graduate.” One possibility is to shift from long-term to peripatetic experts as capacity is successfully built and a certain threshold is surpassed.

(The lack of funding for implementing and following-up TA recommendations became an obstacle to the large impact of TA.) A closer coordination with international donors to provide a “full package” where TA recommendations are matched with resources from these donors would be ideal. Pooling donors’ resources, to make sure ex ante that TA recommendations will be properly funded, is desirable.

2) “Enhancing the Impact of Fund Technical Assistance” prepared by the Office of Technical Assistance Management (In consultation with other Departments), April 3, 2008

81. This paper was prepared with the view to make Fund TA more effective, ensure that TA resources are allocated efficiently, and that TA is better coordinated with recipient countries, other providers, and development partners. The Medium-term Strategy and Independent Evaluation Office (IEO) recommendations have called for substantial changes in the way the Fund conducts business, including information management and disclosure. There is a supplemental document to this paper titled: Dissemination of Technical Assistance Information (SM/08/97). The main issues discussed in this paper for substantial reforms (in prioritizing, monitoring, evaluating, and costing TA) are summarized as follows:

Enhance the integration of TA with surveillance and Fund lending operations. Enhance prioritization by better aligning TA with strategic objectives of the recipient

country and coordinating with other TA providers. Make the TA more responsive to changes in priorities. Better integrate TA into the Fund’s medium-term budget to make priority-setting easier. Introduce Fund-wide performance indicators (moving to results-focused management

of TA projects) to make TA more transparent and accountable. Refine the costing and billing of TA projects to help ensure efficient allocation of

resources, better inform management decisions and enhance accountability. Strengthen partnerships with donors with a more proactive external financing strategy.

3) “Evaluation of the Policy and Human Resources Development (PHRD) Trust Fund” was prepared by UNIVERSALIA, January 2008.

82. The PHRD trust fund to the World Bank is another technical assistance contribution


11

to an international organization by the Ministry of Finance of Japan. Due to its relevance to JSA, the contents of this report can be of reference for JSA evaluation.

PHRD TA Effectiveness83. Regarding the effectiveness of PHRD TA, the report describes: “The PHRD TA is a highly effective facility that has produced quality results in almost all areas. As an investment, it has led to a high level of outputs and good quality results for the resources expended. PHRD technical assistance is rated highly by both the World Bank and government officials, particularly in terms of the quality of project preparation for Bank loan operations. PHRD grants do not appear to duplicate other Official Development Assistance (ODA) resources.”3

Recommendations84. The report concludes by presenting the following six recommendations4.(1) A Bank-wide discussion, possibly led by the Operations Policy and Country Service

Department (OPCS), should be initiated to develop a position paper on the financing of project preparation in the Bank and to make the experience of PHRD TA more widely known to Bank investors.

(2) Given a future in which the increasing demand for project preparation support may exceed the supply, the World Bank and the Government of Japan will need to establish more strategic priorities for PHRD TA to ensure that it manages available resources in the most cost-effective way. The Recommendation 2 includes, among others; revising allocation between IDA/ non-IDA countries and targeting climate change grants to encourage broader based mitigation projects

(3) The Government of Japan and TFO (Trust Fund Operation) should identify tools and processes that support a greater recognition of Japan’s dedicated support to good project preparation through PHRD TA. The Recommendation 3 includes: symbolic changes such as putting the Japanese flag on PHRD TA documents; Process changes to enhance recognition of Japan as the sponsor of TA; Strategic changes for more country-level coordination between WB and the Japanese Embassy at the selection stage.

(4) The TFO and the Government of Japan should consider limited additional flexibility in PHRD TA grant conditions in order to increase the effectiveness of the grant program.

(5) The World Bank and Government of Japan should consider ways to increase efficiency of the PHRD TA grant approval process, to avoid unnecessary delays in a process that can take six months to a year or more to complete.

(6) TFO should ensure greater oversight of the PHRD TA reporting system to check that GRM (Grant Reporting and Monitoring) reports are complete and that they provide adequate data to monitor and assess the efficiency and effectiveness of the PHRD TA grants.

3 Page xiii, Final PHRD Synthesis Report, UNIVERSALIA, January 20084 Quoted from page xv to xvii of Final PHRD Synthesis Report, UNIVERSALIA, January 2008

2. Outline of JSA and Japanese ODA

12

Chapter 2 Outline of JSA and Japanese ODA

2.1 Outline of JSA 2.1.1 Background and Objectives

1) Background 85. The history of JSA dates back to 1990, when Japan started to provide grant contributions to support the IMF’s technical assistance to member countries. Among the external financial sources which supported IMF TA, JSA is one of the major financial sources. In fact, since 1990, Japan has been the largest contributor to the IMF’s technical assistance (TA) activities.5 In FY2009, external financing from bilateral and multi-lateral donor partners occupied close to 70 percent of technical assistance delivered in the field, and the rest was sourced from IMF internal financing. Despite the trend of increasing financial sources from bilateral and multilateral donors, the Japanese share among the external financing source remains the largest. In FY2009, JSA financing accounted for about one-third of total external sources, which plays a dominant role in supporting the IMF TA. Chart 2.1 shows the historical trend of JSA’s share of financing of the IMF’s field-delivered technical assistance from FY2000 to FY2009.

Chart 2.1 JSA Share of Financing of Technical Assistance, FY2000–FY2009

0

10

20

30

40

50

60

70

FY1997 99 01 03 05 07 09

Per

cent

IMF

JSA

Others

(Source: JSA Annual Report 2009 and others)

5 Contributors to IMF TA other than Japan include Australia, Austria, Belgium, Canada, China, Denmark, Finland, France, Germany, India, Ireland, Italy, Korea, Kuwait, Luxembourg, the Netherlands, New Zealand, Norway, Oman, Portugal, Qatar, Russia, Saudi Arabia, Spain, Sudan, Sweden, Switzerland, the United Arab Emirates, the United Kingdom, and the United States. (In addition, Singapore, Brazil, the Arab Monetary Fund and the African Development Bank provide support for IMF Institute’s regional training centers.)


13

86. Chart 2.2 shows the historical trend of annual contributions which were used solely for the “technical assistance” portion of the JSA since its foundation in 1990.6 The figure indicates that since 1990 Japan has made annual contributions for JSA totaling $305 million, out of which $278.9 million was used for TA projects including the activities of the Regional Office for Asia and the Pacific.

Chart 2.2 Annual Contributions by Japan for Technical Assistance FY1990–FY2009

(Source: JSA Annual Report 2009)

87. The detailed breakdown of Japanese contributions and JSA is also indicated in Chart 2.3.

Chart 2.3 Contributions by Japan FY1990–FY2009(In millions of U.S. dollars)

FY1990-2000

FY2001

FY2002

FY2003

FY2004

FY2005

FY2006

FY2007

FY2008

FY2009

Total FY1990- FY2009

JSA 138.5 15.1 25 25.5 20.4 20.8 22.1 19.3 18.2 20.9 325.9

Technical Assistance 131.9 13.7 22.6 22.8 17.6 18.3 19.4 16.9 15.6 18.6 297.5

Asia Scholarship Program 6.6 1.4 2.3 2.7 2.7 2.6 2.7 2.4 2.6 2.3 28.4

Advanced Scholarship 5.6 1.4 1.4 1.5 1.5 1.5 1.6 1.5 1.5 1.6 19.2

Note: “Technical Assistance” includes activities of the Regional Office for Asia and the Pacific. (Source: JSA Annual Report 2009)

6 JSA supports the activities implemented by the IMF Regional Office for Asia and the Pacific (OAP) in Tokyo and also the Japan-IMF Scholarship Program for Asia (JISPA), which is administered by OAP. This evaluation is limited to TA activities financed by the JSA, and does not cover OAP or the scholarship program.

0.0

5.0

10.0

15.0

20.0

25.0

FY90 92 94 96 98 2000 02 04 06 08

US$ Million


14

2) Objectives

88. The main objective of JSA TA has been to provide assistance in institutional and capacity building for low-income countries and lower-middle-income countries to sustain their economic growth. Regarding the target area of support, the use of JSA generally reflects the priorities of IMF technical assistance as a whole. JSA TA has been targeted mainly in the monetary and financial area, fiscal and budget management, Anti-Money Laundering (AML) and Combating the Financing of Terrorism (CFT).

89. As for the usage of funds, JSA funds can be used to cover the cost of short- and long-term TA experts and other costs associated with conducting seminars and workshops. The selection of TA experts is based on quality in international best practices depending on the requested subject of TA. Regarding geographic coverage, the JSA funds can be used in countries all over the world with no restrictions, but high priority is placed on TA activities in Asia and the Pacific region as well as Central Asia. In this regard, there is a general guideline within the IMF that about half of JSA funds should be used in these regions.

90. The Japanese authorities also place high priority on assistance for countries that have demonstrated strong effort and good track records in the implementation of economic reform policies.7

2.1.2 Institutional Set-up at IMF

91. In providing JSA TA, along with other IMF TA, the functional departments, such as the Fiscal Affairs Department (FAD), Monetary and Capital Markets Department (MCM), and Statistics Department (STA) play a substantial role in project formulation and selection of experts, while area departments play a key role at the initial stage in needs identification. The Office of Technical Assistance Management (OTM) coordinates JSA policy and procedures in consultation with other IMF departments, such as various area departments and the Japan Executive Director’s Office. OTM also functions as a window in communicating with Japanese authorities.

1) Preparation and Formulations Process of JSA projects

92. Projects to be funded from the JSA as well as other IMF TA are planned and prepared within an annual cycle. The next Chart 2.4 shows the preparation process within IMF.

7 This is indicated in the paragraph “Technical Assistance Funded by the JSA” on page 7 of “Japan Administered Account for Selected IMF Activities (JSA) ANNUAL REPORT Fiscal Year 2009”.


15

Chart 2.4 Workflow of Preparation and Formulation of JSA Projects

(Source: MURC based on interviews at IMF Headquarters)

93. First, Area Departments assume their role to identify needs in areas and countries, considering the results of surveillance activities and TA needs. The Regional Strategy Note (RSN) compiled by each Area Department is a basic document which outlines regional issues to be addressed, assistance policy and plan in the region and respective countries, and future direction. Then, Functional Departments review needs of recipient countries and set priorities considering RSN, and prepare another key document called the Resource Allocation Plan (RAP), which presents the forecasts of the regional distribution of both

Area Departments (area specialists)-Prepares Regional Strategy Notes (RSN), which reflects policies of each mission chief

Functional Departments (FAD, MCM, Statistics, etc) (subject specialists) -Review needs of recipients countries and set priorities considering RSN -Prepares Resource Allocation Plan (RAP)-Considers the fund source of each possible project (JSA is considered to be the most reliable and flexible fund source among external financing.)

Area Departments -Receives feedback and endorse

Assistance request from member countries

Functional Departments (FAD, MCM, Statistics, etc)Regarding the requested projects; - Conduct Diagnostic Study: identify issues to be addressed and consider countermeasures to be taken by IMF. Donor coordination is done to avoid duplications. Resident Representatives and experts support the mission depending upon needs. - Prepare Project Proposal: Scope of works, target, implementation period, mode of assistance are proposed. Background and necessity, goals are explained.

Committee on Capacity Building (CCB) reviews proposal

Office of Technical Assistance Management (OTM): -Check the conformity with JSA guidelines

-Prepares indicative list of JSA candidate projects

Japan Executive Director’s Office: -Receives indicative list of JSA

Ministry of Finance, Japan: - Receives indicative list of JSA and gives approval


16

internally- and externally financed TA, including JSA resources. The RAP is agreed upon after extensive discussions among Area Departments, Functional Departments, and OTM. Through these documentation and agreement processes of RSN and RAP, the needs and priorities for TA in each country are reviewed and discussed.

94. At the beginning of each fiscal year, IMF provides Japan’s Ministry of Finance with an indicative list of projects which are planned to be submitted during the year for consideration. Thereafter, individual projects are submitted for approval on a monthly basis through the Office of Japan’s Executive Director at the IMF.

2) Project Assessment and Monitoring

95. After the project is completed, project managers at IMF TA-providing departments prepare a project assessment report. In addition, project managers collect the recipient’s evaluation of JSA projects using simple questionnaire sheets distributed to recipients in order to extract lessons for further improvement of IMF TA for the future.

96. Moreover, a joint Japan-IMF mission for JSA projects is organized, and two or three selected countries are visited in order to review projects every year. Japan’s MOF receives first-hand information from these field visits as to whether JSA funds are effectively used in each country. Also, consultation meetings between IMF and MOF are held occasionally (semi-annually, as defined in the guideline 8 ) where they discuss agendas such as effectiveness of JSA, expected use of JSA resources, the plan of joint field visits and other relevant issues.

2.1.3. Record of Past Activities

1) Transition of JSA amount (commitments and disbursements)

97. FY1993 to FY2009: As Chart 2.5 “Annual Commitment and Disbursement of JSA TA, FY1993-FY2009” indicates, between FY1993 and FY2009, a cumulative commitment for technical assistance under the JSA was $262.5 million for 1,852 projects. Out of this committed amount, $251.9 million has been disbursed.

98. FY2004 to FY2008: Between FY2004 to FY2008, as Chart 2.6 “Annual Commitment and Disbursement of JSA TA, FY2004-FY2008” shows, a cumulative sum of $88.4 million was committed for 560 projects in total, and cumulative $90.9 million was disbursed.

8 Guidelines for the Use of Resources for Technical Assistance Activities from The Administered Account for Selected Fund Activities-Japan” issued in July 2006.


17

2) Geographical distribution of JSA funds 99. FY1993–FY2009 To date, JSA covered 124 member countries and 13 Regional TA Centers and Training Institutions. During FY1993–FY2008, countries in the Asia and Pacific region received $90.7 million, or 37% of TA projects approved during this period. Africa received the next largest share, totaling $59.5 million, or 24% of total approvals. The detailed geographical distribution including FY2009 is indicated in Chart 2.7.

Chart 2.5 Annual Commitments and Disbursement of IMF/JSA TA, FY1993-FY2009Number of Projects

Committed Committed

(US$ Million) Disbursed

(US$ Million)

FY1993 68 5.7 2.9 FY1994 98 8.8 7.1 FY1995 143 13.1 12.2 FY1996 128 15.1 13.9 FY1997 116 14.5 15.5 FY1998 96 13.6 10.8 FY1999 112 20.7 16.8 FY2000 106 17.3 18.5 FY2001 110 16.4 15.7 FY2002 103 16.7 14.6 FY2003 108 17.3 16.3 FY2004 114 18.2 19.4 FY2005 119 17.4 18.9 FY2006 120 18.8 19.7 FY2007 122 20.7 19.4 FY2008 85 13.3 13.5 FY2009 104 14.8 16.5

Total 1,852 262.5 251.9


Chart 2.6 Total Commitments and Disbursement of IMF/JSA TA, FY2004-FY2008

FY Number of Projects Total committed Total disbursed

2004-2008 560 US$88.4 million US$90.9 million


Chart 2.7 IMF/JSA Commitments for Technical Assistance by Region, FY1993–FY2009(Million US dollars)

FY1993-2000

FY2001

FY2002

FY2003

FY2004

FY2005

FY2006

FY2007

FY2008

FY1993- FY2008 FY2009

Total Share Total Share Africa 20.7 4.1 4.8 4.9 5.1 4.8 4.6 6 4.7 59.5 24% 4.2 28% Asia and Pacific 37.1 4.7 6.2 6.6 7 7.6 7.9 8.3 5.3 90.7 37% 5.9 40% Europe 6 1.4 1.6 1.2 1 0.7 1.4 1.3 0.5 15.1 6% 2 13% Eastern Europe and Central Asia 528.2 2.7 1.4 1.5 1.4 0.6 1.4 1.8 0 38.9 16% 0 0%

Middle East and Central Asia 2.5 0.7 0.4 1.1 1.6 1.2 2.1 2.4 1.7 13.6 5% 2.1 14%

Latin America and Caribbean 6.9 1.1 0.6 0.6 0.6 0.9 1 0.9 0.9 13.4 5% 0.7 5%

Multiple regions 6.8 1.8 1.7 1.3 1.6 1.7 0.5 0.2 0.2 15.7 6% 0 0%

Total 108.1 16.4 16.7 17.3 18.2 17.4 18.8 20.7 13.3 247 100% 14.8 100%



18

100. FY2004 to FY2008: The cumulative results of regional distribution between FY2004 to FY2008 are indicated in Chart 2.8. The total amount committed during the five years was $88.4 million, which was allocated for 560 projects in total, and $90.9 million was actually disbursed.

Chart 2.8 IMF/JSA Commitments for Technical Assistance by Region (FY2004–FY2008)(Millions U.S. dollars)

Region FY 2004 FY 2005 FY 2006 FY2007 FY2008 FY2004- 2008

FY2004- 2008 (%)

Africa 5.1 4.8 4.6 6.0 4.7 25.2 28.3% Asia and Pacific 7.0 7.6 7.9 8.3 5.3 36.1 40.6% Europe 1.0 0.7 1.4 1.3 0.5 4.9 5.5% Eastern Europe and Central Asia

1.4 0.6 1.4 1.8 - 5.2 5.8%

Middle East and Central Asia

1.6 1.2 2.1 2.4 1.7 9.0 10.1%

Latin America and Caribbean

0.6 0.9 1.0 0.9 0.9 4.3 4.8%

Multiple regions 1.6 1.7 0.5 0.2 0.2 4.2 4.7% Total 18.2 17.4 18.8 20.7 13.3 88.4 100.0%


101. In this period, Asia and Pacific received $36.1 million, or 40.6% of the total, which is higher than the long-term trend. Ranked second was Africa, which received a total $25.2 million, or 28.3% share. The share of geographic distribution from FY2004 to FY2008 is shown in Chart 2.9.

102. In the recent trend over a five year period (2004-2008), the Asia and Pacific region continues to be the largest recipient of JSA. The combined share of ‘Asia and Pacific’ and ‘Africa’ was 68.9% from 2004 and 2008. Considering the fact that the combined share was 61% from 1993 to 2008, it is noted that these regions are given more priorities in recent years within the total portfolio of JSA.


19

Chart 2.9 Percentage Share of Cumulative Commitment by Region FY2004-FY2008

Latin America andCaribbean

4.8%

Multiple regions4.7%

Middle East and CentralAsia

10.1%

Eastern Europe andCentral Asia

5.8%

Europe5.5%

Asia and Pacific40.6%

Africa28.3%

(Source: MURC based on the data of JSA Annual Report 2009)

3) Distribution of JSA Funds by Subject Area

103. FY1993–FY2009: As Chart 2.10 shows, during FY1993–FY2008, JSA funds were spent on the Monetary and Capital Markets area, which received a total of $95.5 million, or 39%, the largest share among six subject areas. The second largest area was Fiscal, with the amount totaling $66.6 million, or a 27% share. Then Macroeconomic Statistics follows with $41.1 million (17%) and Training with $28.8 million (12%). For FY2009 only, the Fiscal area received $5.8 million, the largest share of 40%, and Monetary and Capital Markets came in second with $4.0 million and 27%.

Chart 2.10 JSA Annual Commitments for Technical Assistance by Subject Area FY1993–FY2009 (Million U.S. dollars)

FY1993–2000

FY2001

FY2002

FY2003

FY2004

FY2005

FY2006

FY2007

FY2008

FY1993- FY2008

FY2009

Total Share Total Share

Fiscal 31.4 4.7 4.7 3.3 4.3 4.3 4.5 6.0 3.5 66.6 27% 5.8 40% Monetary and capital markets

41.6 6.5 6.6 6.7 6.9 6.6 7.8 8.2 4.7 95.5 39% 4.0 27%

Macroeconomic statistics

16.8 2.7 2.7 3.7 3.3 2.8 3.1 3.7 2.3 41.1 17% 2.8 19%

Training 12.6 1.9 2.1 2.1 2.2 2.0 2.0 2.1 1.8 28.8 12% 2.0 13% Legal 1.6 0.1 0.2 1.2 1.2 1.2 0.9 0.7 1.0 8.2 3% 0.2 1% Others 4.1 0.6 0.5 0.3 0.2 0.6 0.6 0.0 0.0 6.9 3% 0.1 0% Total 108.1 16.4 16.7 17.3 18.2 17.4 18.8 20.7 13.3 247.1 100

%14.8 100

%

(JSA Annual Report 2009)


20

104. FY2004 to FY2008: Chart 2.11 indicates the trend of distribution of committed amount by subject areas between FY2004 to FY2008.

Chart 2.11 JSA Commitments for Technical Assistance by Subject (FY2004–FY2008)(Millions U.S. dollars)

Subject FY 2004 FY 2005 FY 2006 FY2007 FY2008 FY2004- 2008

FY2004- 2008 (%)

Fiscal 4.3 4.3 4.5 6.0 3.5 22.6 25.5% Monetary and Capital Markets

6.9 6.6 7.8 8.2 4.7 34.2 38.6%

Macroeconomic statistics

3.3 2.8 3.1 3.7 2.3 15.2 17.2%

Training 2.2 2.0 2.0 2.1 1.8 10.1 11.4% Legal 1.2 1.2 0.9 0.7 1.0 5 5.6% Others 0.2 0.6 0.6 0.0 0.0 1.4 1.6%

Total 18.2 17.4 18.8 20.7 13.3 88.4 100.0%


105. When we compare the long-term trend of sixteen years (FY1993-FY2008) and the most recent structure, the notable difference is that Fiscal has now become the priority area rather than the Monetary and Capital Markets. The current share of Fiscal in 2009 is 40%, which is much higher than the 27% recorded by Monetary and Capital markets. With ‘Fiscal’ and ‘Monetary and Capital Markets’ combined together, these two subject areas have constantly stayed as the major recipient, which received 64.1% of the total amount committed during FY2004 to 2008, and 67% in FY2009. The detailed percentage share of subject area distribution from FY2004 to FY2008 is shown in Chart 2.12.

Chart 2.12 Percentage Share of Cumulative Commitment by Subject FY2004-FY2008

(Source: MURC based on the data of JSA Annual Report 2009)


21

106. As this graph shows, in the recent five years, Monetary and Capital Markets occupies the largest share of 38%, which is almost the same as the long-term trend of 39% between FY1993 and FY2008. Ranking second is Fiscal with 26%, again almost the same level as the long-term trend. After that, Macroeconomic statistics follows with 17%, and then Training and Legal with 11% and 6% respectively.

2.2 ODA Policy of Japan

107. The Japanese official development assistance (ODA) is structured as indicated in Chart 2.13. The modes of assistance are classified into two major categories; (A) Direct cooperation with recipient countries, and (B) Cooperation through International Organizations such as IMF, the World Bank and United Nations. The support to JSA falls under category (B). In other words, direct cooperation with recipient countries is equivalent to “bilateral” assistance, and cooperation through international organizations is equivalent to “multilateral” assistance.

Chart 2.13 Classifications of Japanese ODA

(Source: Japan’s International Cooperation, Ministry of Foreign Affairs)

108. The direct cooperation with recipient countries is further divided into (a) Bilateral Donation, and (b) Government Loan (which means “Yen Loan” available through JICA)9.The Bilateral Donation consists of (i) technical cooperation and (ii) grant aid. When the Japanese Government sends experts overseas, this is included in this technical cooperation category.

109. The historical trend of Japanese ODA on a net disbursement basis from 2004 to 2008 is indicated in Chart 2.14.

9 Japanese ODA loan is only by Japanese Yen currency.


22

Chart 2.14 Trend of Japanese ODA: Net Disbursement Amount By Type of Assistance 2004-2008 (Calendar year)

(Million US Dollars)

Type 2004 2005 2006 2007 2008 10

Bilateral Donation 7,235 9,274 7,734 6,046 5,556

Technical Cooperation 2,909 2,749 2,684 2,630 3,614

Grant Aid 4,326 6,525 5,050 3,416 1,941

Loans 11 (1,281) 1,210 (251) (206) 2,277

Contribution to International Organizations

3,048 2,799 3,878 1,907 2,557

Total Net Disbursement 9,002 13,283 11,361 7,747 10,389

(Source: ODA White Papers, 2004 to 2008, Ministry of Foreign Affairs of Japan)

110. Technical Cooperation has been provided with a size of $2.6 to $2.9 billion by various ministries and agencies between 2004 and 2007, on an actual net disbursement basis. The contribution to International Organizations has been fluctuating, but the amount contributed to JSA itself has been relatively stable with around US$20 million or so, which was previously indicated in Chart 2.3. Loans have been a major form of Japanese ODA which occupies around half of the total ODA budget, if counted on a gross basis.

111. In looking at historical trends of Japanese ODA amounts, it should be noted that Japanese ODA has different fund sources including; (1) general account, (2) special account, (3) issuance of government bonds, and (4) fiscal loan and investment. Loans, for example, are mostly sourced from Fiscal loan and investment. Regarding the sources of technical cooperation, in the FY2009 ODA budget plan, 84% of the total amount comes from the general account, 11% from fiscal loan and investment, and 5% from the special account. This indicates the importance and impact of the general account. The historical trend in general account based ODA budget is shown in Chart 2.15.

112. As Chart 2.15 indicates, ODA budget sourced from the General Account peaked in 1997, and it has been decreasing sharply over the years. The rate of decrease reached around 40% in the eleven years from 1997 to 2008. Given the current tight budget situation in Japan, this decreasing trend in the ODA budget from the general account is forecasted to continue.

10 The figure in 2008 is “budget” amount on net disbursement, and not actual yet. 11 This “Loan” category is expressed on “net” basis. The repayment and collection from borrowers are counted.


23

Chart 2.15 Japanese ODA Budget Sourced from General Account

(Source: Ministry of Foreign Affairs)

2.3 Consistency of IMF/JSA TA and Japanese ODA


113. Japanese ODA policies are represented by Japan’s Official Development Assistance Charter, ODA Mid-term Policy and Country Assistance Programs. The initial version of the ODA Charter was approved by the Cabinet in 1992, and it served as the foundation of Japan's aid policy for more than 10 years. In August 2003, it was revised to reflect the changing environment and the new challenges of the international community, and has guided the Japanese assistance strategy to the present. In relation to technical assistance provided by IMF, the following points of the ODA Charter are highly relevant and consistent; (1) support to self-help efforts of recipient countries, (2) cooperation with human resource development, and institution building in the legal and economic area, and (3) promotion of cooperation with international organizations. The details of these relevant provisions are shown in the table below.


24

Chart 2.16 Relevant Clause of Japanese ODA CharterItem Description

Basic Policy No.1

Japan will carry out ODA even more strategically, in accordance with the following basic policies. (1) Supporting self-help efforts of developing countries The most important philosophy of Japan's ODA is to support the self-help efforts of developing countries based on good governance, by extending cooperation for their human resource development, institution building including development of legal systems, and economic and social infrastructure building, which constitute the basis for these countries’ development. Accordingly, Japan respects the ownership by developing countries, and places priorities on their own development strategies. In carrying out the above policy, Japan will give priority to assisting developing countries that make active efforts to pursue peace, democratization, and the protection of human rights, as well as structural reform in the economic and social spheres.

Basic Policy

No.5

(5) Partnership and collaboration with the international community Mainly with the initiative of international organizations, the international community is sharing more common development goals and strategies and various stakeholders are increasingly coordinating their aid activities. Japan will participate in this process, and endeavor to play a leading role. In parallel with such efforts, Japan will pursue collaboration with United Nations organizations, international financial institutions, other donor countries, NGOs, private companies, and other entities. In particular, Japan will enhance collaboration with international organizations that possess expertise and political neutrality, and will endeavor to ensure that Japan's policies are reflected appropriately in the management of those organizations.

(Source: Japan’s ODA Charter, Ministry of Foreign Affairs)

114. Next, with respect to ODA Mid-term Policy which has been established since 2005, again, the IMF TA has high relevance. There are three major relevant mid-term policies which are consistent with the philosophy of IMF TA: (1) Macroeconomic stabilization through appropriate fiscal and monetary policy as a prevention measure for poverty, (2) Policy formulation and institution building necessary for sustainable growth, and (3) Assistance in human resources development to promote sustainable growth. The detailed description of these points is highlighted below.

Chart 2.17 Relevant Clause of Japanese ODA Mid-Term PolicyItem Description

3. Priority Issues, (1) Poverty reduction

(d) Assistance for institutions and policies to reduce poverty (iii) In order to minimize the impact on the poor during economic crises, inflation and similar events, macroeconomic stabilization through appropriate fiscal and monetary policy is essential. To this end, assistance such as dispatching experts will be provided to build the capacity of government officials.

3. Priority Issues, (2) Sustainable growth

(b) Policy formulation and institution building To promote macroeconomic stability, Japan will assist in the formulation andimplementation of appropriate and sustainable fiscal and monetary policy, publicdebt management, and economic policy, and will place an emphasis on assisting the formulation of industrial policy designed to expand trade and investment, and of rural and regional development policy in the light of decentralization. In concrete terms, assistance will be provided for institution building in the fields ofeconomic management, finance, tax, customs and the development of human


25

resources, and the development of local and supporting industries. To developing countries that are in transition to a market economy, particular support will be provided for policy formulation, institution building, development of legal systems, and the fostering of human resources to facilitate such transition. Eradicating corruption, implementing legal and institutional reforms, improving the efficiency and transparency of public administration, and strengthening the administrative capacity of local government are important to building a fair and democratic society and also to improving the investment climate. For this purpose, Japan will assist the capacity-building of governments to improve governance.

(Source: Japan’s ODA Medium-Term Policy, Ministry of Foreign Affairs)

115. In addition to the general ODA policies introduced above, Country Assistance Programs (CAP's) are determined for each country based on the ODA Charter and Mid-term Policy. Since CAP's are completely bound by the philosophy and directions of these two upstream strategy documents, we can construe that activities of IMF TA are also in line with the priorities of CAP's. Since it is difficult to compare IMF TA with all the CAP’s issued, the CAP’s of Indonesia, Cambodia and Vietnam have been verified. The results show that IMF TA is highly consistent with each CAP as shown in the chart below.

Chart 2.18 Relevant Clause of Japanese Country Assistance Programs (CAP’s)Country Relevant Description in CAP’s

Indonesia CAP in 2004 maintains that, to ensure “Fiscal Sustainability” Japan will provide assistance such as policy advice and human resource training for the creation of a structure and system for fiscal reform such as strengthening the annual government revenue, reinforcing development planning functions and appropriate management of government expenditures based on a well-considered development plan. Also, special mention is made for support of “Reforming the Financial Sector”; Governance of financial institutions and risk screening abilities are inadequate in Indonesia, and financial intermediation functions are weak. In addition, the capital market is not yet developed in terms of the scale of issuance and the scale of liquidity, preventing the private-sector companies from raising the capital they need to energize their investment activities. The Indonesian government is engaged in I) strengthening the financial structure of financial institutions, ii) setting up supervisory functions in the financial sector, iii) fostering SME financing, iv) arranging for trade financing and v) cultivating a sound capital market. Japan will assist the Indonesian government’s efforts in these areas by providing assistance in building systems and developing human resources.

Cambodia CAP in 2002 states that Japan will promote and support particularly the government’s efforts in reform and good governance in five areas; (1)administration, (2)fiscal management, (3)reduction of military forces, (4)management of natural resources, and (5)social sector.

Vietnam CAP in 2004 point out that institutional building, development of the legal system and administrative reform are the priorities. Assistance for judicial reform is emphasized, and for support of administrative reform, it defines that “Priority for assistance is given toreform of the civil service system and fiscal reform (public expenditure management, including taxation and the taxation system, and enhancing links between policies and resource allocation mechanism).

(Source: Japan’s Country Assistance Programs, Ministry of Foreign Affairs)


26

116. From these observations, IMF TA through JSA can be considered highly consistent with Japanese ODA policies.

2) Duplication with Other Technical Assistance Activities of Japan

117. Japanese Technical Assistance is provided in the range of around $2,600 to $2,900 million every year by the entire Japanese government. It is mainly directed to subject areas such as transportation, agriculture, health and medical care and education, but the experts sent in the fiscal, monetary and financial field are rather limited. The field survey results indicated JSA and TA by other donors were complementary. There was no duplication between the IMF/JSA TA and other Japanese ODA initiatives. IMF/JSA TA and other Japanese TA activities do not overlap and they play different roles, or complement each other even in the same sector of the same country. For example, in Indonesia, TA for banking supervision at Bank Indonesia (BI) was dominated by IMF experts. JICA did not send any expert to BI and there was no duplication. In Vietnam, duplication of TA was intentionally avoided by coordinating efforts among donors.

3. Evaluation Results through Data Analysis and Questionnaires

27

Chapter 3 Evaluation Results through Data Analysis and Questionnaires

3.1 Review of Evaluation Reporting System of JSA

3.1.1 Project Assessment for JSA

118. Based on the JSA Guideline (2006), the TA-providing departments are to submit a written assessment of each IMF/JSA TA project in the specific format within four weeks of project completion. According to the OTM, in practice, project assessments have been prepared and submitted for all IMF/JSA TA projects.

119. In the project assessment, the current template requires the following information:

General Information: Project Title, Project Number, Country, Fiscal Year, Lead Dept., Area Dept., Project Data, Project Manager, Local Counterpart, and Budget

Project Description Overview of Progress Achieved (Project Objectives, Project Outcomes)

- Project Objectives: Objectives, Verifiable Indicators, Achievement (1-4 Scale), and Implementation Status - Project Outcomes: Priority, Outcomes Description, Completion Date, Achievement (1-4 Scale), and Comment on Achievements

Notes for Future Projects

120. In the evaluation study, these “Project Assessments” for IMF/JSA TA projects have been reviewed for meta-evaluation purposes. The reviews are carried out in light of the following two view points: 1) comprehensive evaluation of assessed projects, and 2) evaluation of the assessment system itself. In order to obtain deeper insight from the review, target project assessments are those conducted in our field survey countries, i.e., Cambodia, Vietnam, Indonesia, and African countries (Tanzania, and Eastern AFRITAC).

1) Comprehensive evaluation of assessed projects

(1) Preliminary Review 121. In the Project Assessments, there are various structures in terms of assessment of results, as shown in the table below. Recent TA projects seem to use the ““Project Objectives” & “Project Outcomes” structure, while older projects seem to use only the “Project Objectives” structure or “(Project) Outputs” structure. There are some projects with more than one “objective” In addition, for some projects, there are sub-categories under certain components such as “Project Objectives,” “Project Outcomes,” or “Outputs.” During the target period of this evaluation study, the majority of focus is on the “Overall Objective,” and “Objective” & “Output” for the target 2 countries.


28

Chart 3.1 Structures of Results Assessment (Unit: number of projects)

Structures Cambodia Vietnam “Project Objectives” & “Project Outcomes” 4 0

“Overall Objective,” “Objective” & “Output”

7 4

“Objectives” & “Outputs” 1 0

Only “Project Objectives” 1 2 Only “(project) Outputs” (“Objectives” are described, but not assessed)

3 0

Total Number of Projects 16 6

Chart 3.2 Numbers of Structure Components Assessed(Including Sub-Components)

(Unit: number of components)

Structure Components Cambodia(16 projects)

Vietnam (6 projects)

“Project Objectives” 17 15 “Overall Objective” 7 4 “Objective” 14 6 “Project Outcomes” 20 0 “(Project) Outputs” 71 25

(2) Methodology for Trial Evaluation (Scoring) 122. For target TA projects of the target countries, all assessed components (“Project Objectives,” ”Overall Objective,” ”Objective,” “Project Outcomes,” and “(Project) Outputs”) were analyzed and scored separately. Specifically, their achievements were analyzed based on descriptions (explanations) shown in these columns, and 4-scale scores. The criteria for the scoring are as follows: 4 points: fully achieved 3 points: largely achieved 2 points: partially achieved 1 point: not achieved

123. For the recent projects, the same 4-scale scoring is ranked in regard to their achievements. There are, however, inconsistencies between those scores and narrative descriptions, so only the latter are used in this scoring. In addition, in some projects, TOR's have changed to some extent, and those are categorized as “un-scored.”

124. Due to the rather judgmental nature of this trial rating and to some inconsistent


29

assessment practices discussed below, the quantitative data of this trial scoring cannot be considered as objective evidence of project achievements.

(3) Results 125. The results of the trial scoring indicate that most of the IMF/JSA TA projects have at least to some extent achieved the proposed targets in regard to their objectives, outcomes, and outputs. There does not seem to be a common trend in the degree of achievements by country. In addition, there are slight differences in evaluating the degree of achievement between the scoring results here and results from the questionnaire survey for beneficiaries and experts (See Chapter 3.3) and JSA Evaluation Questionnaire for beneficiary authorities (See 3.1.2). In the results from those questionnaire surveys, respondents have almost unanimously answered IMF/JSA TA projects as effective (targets achieved). The results of this trial scoring, however, indicate that degrees of achievement are relatively varying. This does not necessarily show real differences between them and might be caused by some judgmental nature of the trial scoring and variation of implementing practices. In addition, the fact that these assessments have been conducted both in detail and internally might also cause variation of assessments.

Chart 3.3 Results of Trial Scoring by ComponentsCambodia: 16 projects

(Unit: number of components) Structure

Components Total

Assessed 4 points 3 points 2 points 1 point

“Project Objectives” 17 8 3 6 0 “Overall Objective” 7 2 4 1 0 “Objective” 14 8 3 2 1 “Project Outcomes” 20 4 8 8 0 “(Project) Outputs” 71 26 19 17 7

(unscored:2)

Vietnam: 6 projects (Unit: number of components)

Structure Components

Total Assessed 4 points 3 points 2 points 1 point

“Project Objectives” 15 4 5 3 3 “Overall Objective” 4 1 1 1 1 “Objective” 6 1 3 1 1 “Project Outcomes” 0 0 0 0 0 “(Project) Outputs” 25 7 9 3 6


30

2) Evaluation of assessment system

(1) Evaluation Skills & Practices 126. Based on findings from the review (trial rating), the project assessment system for the IMF/JSA TA projects seems to have some room for improvement from the viewpoints of better accountability and project management. The following are major findings:

(2) Inconsistent Explanation About Achievements 127. There is a general tendency that description methods and styles of project achievements (“Project Objectives,” ”Overall Objective,” ”Objective,” “Project Outcomes,” and “(Project) Outputs”) vary and are inconsistent from project to project. In some cases, negative descriptions are used for indicating the necessity of further assistance, rather than assessing projects in light of project objectives and targets. These seem to be caused by the non-existence of shared common criteria (standards) for the assessment practice.

(Example) Project Objectives: To further improve government xxxx. Achievement: 2

Whereas for most xxx, the new xxx was appropriately used, xxx was incomplete.

Project Objectives: Achievement: 4 Objective achieved. XXX have as yet not been xxxed due to XXX. XXX will need to be trained in XXX areas.

(3) Mixture (or Inconsistent Use) of “Outputs” and “Outcomes” 128. There is some inconsistency between “outputs” and “outcomes.” In addition, in terms of “outcomes,” some of them are immediate outcomes while others are intermediate or the final outcomes.

(Examples) Project Objectives: The advisor will provide xxx training. Project Objectives: Provide day-to-day advice ………These are outputs.

Overall Objective: Increase revenue … …..... This is final outcome. Outcomes: Assist the xxx in strengthening the xxx ……..This is an output. Outcomes: With the advisor’s assistance, the xxx staff will produce xxx

reports. Outcomes: Using the new xxx. ……… These are immediate outcomes.


31

Outcomes: Adequate legal framework. Outcomes: Improved xxx management. ……… These are intermediate

outcomes.

(4) Lengthy and Narrative Indicators 129. Many descriptions in the “verifiable indicators” column are lengthy and difficult to be included as appropriate indicators for clearly measuring project progress or achievement.

(5) Ambiguous Project Objectives & Outcomes 130. Although the matrix titles are “Project Objectives” and “Project Outcomes“, the “mid-term objectives and outcomes” are used in some cases.

131. Achievements are written in view of the mid-term objectives rather than one-year project objectives and this causes difficulty in determining the achievements for each project.

(Example) Project Objectives: To modernize xxxxx administration Achievement: 2 Modernization of xxx is an ongoing process. While good progress was made in the key objectives in the 200-2008 plan, there is still considerable work in progress.

(6) Abstract Assumptions 132. In some cases, the assumptions shown in the matrix are general and abstract, and not specific to the project for a particular year.

(Example) Cooperation and support of the authorities, dedicated counterparts, and financing and technical assistance required.

(7) Weakness in Theoretical Models 133. As a result of the major findings above, the current practices in project assessments show weakness in the theoretical models of project management as well as project evaluation.

(8) Project (Program) Management 134. Although performance measurement for a project is a major component of the PDCA cycle for better project (and program) management, practices in IMF/JSA TA projects seem to have some necessity for improvement. Situations the evaluation team faces this


32

time indicate that there are issues within project management practices, in particular, in implementing and accumulating project assessments. For the latter issue, the establishment of an accessible database is a factor for the more active use of evaluation information for better project management. In addition, those databases are a premise for the further evaluation of the IMF/JSA TA projects.

3.1.2 “JSA Evaluation Questionnaire” of IMF

135. The survey “JSA Evaluation Questionnaire” was conducted based on the “JSA Guidelines” (2006). According to these guidelines, “Within five weeks of the completion of a project, OTM is to send the evaluation questionnaire to the person indicated on the TA-department’s assessment report on completed projects.” According to the OTM, around a third of relevant authorities have responded to the questionnaires.

136. In this evaluation study report, results of the “JSA Evaluation Questionnaires” that are answered by beneficiary authorities upon completion of each IMF/JSA TA project are reviewed also for meta-evaluation purposes. Just as above, the reviews are carried out in light of the following two view points: 1) comprehensive evaluation of evaluated projects, and 2) evaluation of the survey system itself. Unfortunately, the number of available evaluation results provided by the OTM for this evaluation study are relatively limited (31 JSA projects) due to the relatively low response rate from authorities, but all evaluations are reviewed here.

137. In the evaluation questionnaire, the following 10 questions are asked:

JSA Evaluation Questionnaire Q1: Did the terms of reference for the expert reflect your needs? Q2: Were you consulted regarding the expert’s terms of reference before his/her

arrival? Q3: Were the expert’s qualifications and experience, in your view, appropriate? Q4: Did the expert cooperate well with his counterparts? Q5: Was the expert’s advice useful in supporting reform? Q6: Did the expert pay sufficient attention to training and capacity building? Q7: Were there relevant issues or tasks that are not addressed by the expert? (If

yes, please specify.) Q8: To your knowledge was the expert adequately supervised by Fund

headquarters? Q9: How would you rate the overall progress of this assignment? Q10: Additional comments, if any.

1) Evaluation of Replied Questionnaires

138. Although the numbers of available answers (i.e. projects) are relatively small, those answers indicate that the IMF/JSA TA projects are generally highly evaluated by the


33

beneficiary authorities. As clearly shown in the chart below, in most questions, more than 80% of beneficiary authorities chose the highest favorable answers for the IMF projects. As the questionnaire is consistent with questions on relevance, effectiveness, and appropriateness of processes of the specific projects, these trends in the responses mean that beneficiary authorities basically regard the IMF/JSA TA projects as quite relevant, effective, and appropriate (in process).

139. In some questions, however, authorities have shown a different recognition or understanding from the trends, as highlighted in the table below. For example, answers for Q6 (Did the expert pay sufficient attention to training and capacity building?) are almost equally divided into two views (Yes: 58.1%, Partially: 41.9%). The reasons for the response results are not clear, because of the simple structure of the evaluation format. One of the hypotheses based on results from other methods in our evaluation study is that advisors of the IMF/JSA TA projects, in particular long-term resident advisors, might tend to directly solve policy and institutional problems by themselves rather than doing so by developing capacities of beneficiaries’ human resources in charge. In addition, answers for Q9 (How would you rate the overall progress of this assignment?) show that almost half of respondents (43.3%) have rated the project as “satisfactory,” and not “highly satisfactory.” One of the assumptions on this answer is that these results do not necessarily indicate certain dissatisfaction by the authorities with the TA projects, but indicate their expectation for further IMF TA projects on the same (similar) policy and/or institutional issues.

Chart 3.4 Results of the “JSA Evaluation Questionnaire”Yes

(Q9: Highly satisfactory) Partially

(Q9: Satisfactory) No

(Q9: Unsatisfactory) Q1 31 (100%) 0 0 Q2 25 (83.3%) 3 (10%) 2 (6.7%) Q3 28 (93.3%) 2 (6.7%) 0 Q4 31 (100%) 0 0 Q5 25 (80.6%) 6 (19.4%) 0 Q6 18 (58.1%) 13 (41.9%) 0 Q7 5 (16.7%) 25 (83.3%) Q8 26 (89.7%) 2 (6.9%) 1 (3.4%) Q9 17 (56.7%) 13 (43.3%) 0

2) Evaluation of Survey System

(1) Evaluation Skills & Practices 140. Questions used in the “JSA Evaluation Questionnaire” surveys are quite appropriate in order to obtain the necessary information for understanding the overall results of the IMF TA projects. A questionnaire was composed of the questions on “relevance (TOR, experts),” “effectiveness (usefulness, satisfaction),” & “process (cooperation, attention, backstopping)” on their TA projects. These are appropriate criteria in evaluating TA projects, considering their “logic models” shown in Chapter I of this report.


34

141. Options for each question consisting of just “Yes” and “No,” or of “Yes,” ”No,” ”Partially Yes” were sufficient for beneficiary authorities of recipient countries to answer. In addition, these options definitely meet “MECE (Mutually Exclusive & Collectively Exhaustive)” criteria generally needed for questionnaire types of surveys.

(2) Project (Program) Management 142. Based on situations the IMF/JSA TA evaluation team faces this time, however, there seem to be some substantial issues in utilizing the information (i.e. evaluation results of “JSA Evaluation Questionnaire”) for project management. In particular, implementation and documentation processes could be improved. First of all, the collection of “JSA Evaluation Questionnaire” for the projects under this evaluation period (FY2004-FY2008) has been limited. Numbers of IMF/JSA TA projects with the evaluation sheets and response rates (31 projects, around 30 %) seem insufficient, based on the requirement of the JSA guidelines that all projects must complete the evaluation sheets. Secondly, these evaluation sheets are received by fax in the past, and the electronic (digital) versions of questionnaires are recently introduced at OTM. However, the information from these results would be quite useful, for example, in enhancing accountability for Japanese people even after data aggregation and/or necessary modification, so the more detailed results is desirable to be reported in the JSA Annual Report.

3.2 Efficiency and Cost Considerations

1) Efficiency of IMF TA

143. In the process of comparing JSA with other donors’ TA, several important features of IMF have been observed, which contribute to the efficiency of IMF TA operations. In this sub-chapter, the following three aspects will be examined: the selection procedure for experts, the high specializations of experts, and the backstopping function of headquarters. From the review of these features of IMF, it could generally be construed that IMF TA is carried out within a relatively reasonable cost.

(1) Expert Selection 144. In selecting experts, IMF employs a roster-based selection system. This system seems to require less operational burden than open application procedures which are typically employed by other major aid agencies including JICA and the World Bank. Time and effort spent on open application (such as making public announcements, inviting applications and their evaluation, negotiations and approvals) seem to be greater than the roster-based selection, although the exact cost comparison is difficult to determine. IMF does not use competitive bidding procedures partly because IMF does not regard itself as a donor agency which is obliged to go to public bidding and partly because the JSA budget


35

size is rather small and does not necessarily fit for such public bidding.

(2) High Specializations of Experts 145. The roster system seems suited for IMF TA because of the high specialization and scarcity of experts available for field assignments. The activities of IMF TA require highly skilled experts with sufficient knowledge of international best practices and hands-on experience in specialized areas, such as banking supervision, budget planning and management, treasury, tax system, money and capital markets, statistics, etc. The experts are typically sourced from government and public agencies such as ministries, central banks and tax authorities. The number of such qualified experts are limited, and currently about 1,350 experts in total are registered in the entire roster within IMF. 12 Due to the specialty and scarcity of experts, there have been cases where the same expert is repeatedly dispatched to the same recipient agency, or to similar organizations in other countries (for example, the same banking supervision expert is dispatched to central banks in different countries), once the expert is judged to be competent by IMF headquarters. In terms of cost consideration, this repetitive assignment of the same expert is an advantage for IMF TA, considering the opportunity costs of dispatching a new person as an expert.

(3) Backstopping Function of Headquarters 146. There is also another factor contributing to the high quality and efficiency of TA which differentiates IMF from other donor agencies. IMF has strong “backstopping” and quality control functions, which lead to the efficient implementation of projects. The general features of IMF and other donors are described in Chart 3.5.

Chart 3.5 General Differences between IMF and Other InstitutionsIMF Other Institutions

- Headquarters staff who work as back stoppers are specialist and experts themselves in the subject area - They are able to directly supervise and guide the advisory services, with good quality control. - Strong backstopping function of Headquarters contributes to efficient implementations - Functional departments have strong authority in selecting experts

- Much work is substantially delegated to consultants and experts themselves, which may include quality control of TA. - Headquarter staff are, in many cases, administration specialists and not experts in subject areas.

(Source: MURC compiled from interviews)

147. The headquarters staff spends time on TA delivery which includes backstopping,

12 This information is based on the interview with IMF Money and Capital Markets (MCM) Department on July 14, 2009. The number of registered experts under the MCM roster is 550, out of which 10 experts have Japanese nationality. For the entire roster within IMF, there are 30 Japanese nationals listed.


36

management, and various administrative support. The time spent on these activities has been recorded, but not allocated to the project cost. To cope with this situation, in April 2008, OTM compiled a package of recommendations for reforms in its report “Enhancing the Impact of Fund Technical Assistance”. In order to measure the cost of headquarters staff, it outlined a plan to collect all project-related cost including staff time. This is an important step for allocating the cost of headquarters staff spent on TA delivery and for establishing a proposed charging system. These efforts at IMF can be considered as evidence of commitment for improving cost efficiency. From the above observations, IMF TA is generally implemented in an efficient manner.

2) Cost considerations

(1) Budget Per Project and its Utilization Ratio 148. To take a look at the approximate budget amount per project, both the commitment and disbursed amounts per project were calculated in Chart 3.6 for the past several years.

Chart 3.6 JSA Annual Technical Assistance Commitments and Disbursements FY1993–FY2009

Fiscal Year

Number of Projects

Committed

Committed ($ Million)

Disbursed ($ Million)

Commitment per Project

($ Thousand)

Disbursement per Project

($ Thousand)

Disbursement Ratio (%)

(A) (B) (C) (B)/(A) (C)/(A) (C)/(B)

1993 68 5.7 2.9 83.8 42.6 50.9% 1994 98 8.8 7.1 89.8 72.4 80.7% 1995 143 13.1 12.2 91.6 85.3 93.1% 1996 128 15.1 13.9 118.0 108.6 92.1% 1997 116 14.5 15.5 125.0 133.6 106.9% 1998 96 13.6 10.8 141.7 112.5 79.4% 1999 112 20.7 16.8 184.8 150.0 81.2% 2000 106 17.3 18.5 163.2 174.5 106.9% 2001 110 16.4 15.7 149.1 142.7 95.7% 2002 103 16.7 14.6 162.1 141.7 87.4% 2003 108 17.3 16.3 160.2 150.9 94.2% 2004 114 18.2 19.4 159.6 170.2 106.6% 2005 119 17.4 18.9 146.2 158.8 108.6% 2006 120 18.8 19.7 156.7 164.2 104.8% 2007 122 20.7 19.4 169.7 159.0 93.7% 2008 85 13.3 13.5 156.5 158.8 101.5% 2009 104 14.8 16.5 142.3 158.7 111.5%

Total 1,852 262.5 251.9 141.7 136.0 96.0%



37

149. According to this Chart, the latest figure in FY2009 shows that commitment per project is US$142.3 thousand, and disbursement per project is US$158.7 thousand. Historically, from FY1993 to FY1999, the amount per project showed an increasing trend, and after FY2000, the trend has been relatively stabilized within the range of US$140 to $170 thousand, which seems to be well controlled. In the recent three years, both commitment and disbursement have been declining. In addition, the ratio of budget execution, which is represented by the figure ‘disbursement ratio’, is high with a total average of 96%. This indicates that the planned budget is highly utilized.

150. During the five years from 2004 to 2008, the total number of IMF/JSA TA projects was 560, 112 projects per year on average, and the disbursed cost was $90.9million, $162 thousand per project on average. The fact that the same issues in the same countries are covered in several cases, it would be more efficient to manage to package a series of TA activities into a larger TA program, if the needs and targets are clearly defined. The benefit of programming is to be able to focus on key issues through a selected approach. The programming approach is already being attempted - for example, a proposed project for Fiscal Management in West Africa with a $5 million package of TA.

(2) The Unit Cost Comparison with Bilateral Assistance 151. The IMF has identified the standard cost of sending experts in its “Guidelines for the Use of Resources for Technical Assistance Activities from The Administered Account for Selected Fund Activities-Japan”. According to this guideline issued in July 2006, as the FY2007 Standard Costs, the monthly amount for experts for long-term assignment is US$21,700 and short-term is US$28,600. Thus, the cost for a long-term assignment of six months is US$130,200, and a twelve months assignment is US$260,400. These costs include benefits, which include an accommodation allowance and expatriation travel for the expert.

152. In comparison with Japanese bilateral technical assistance, for example, these costs can be considered to be at a reasonable level. In the case of a JICA long-term expert with a comparable experience level, his or her remuneration rate is about $11,000 per month, not including social insurance and overhead costs. The social insurance, pension-related cost and overhead cost would likely amount to $7,000 to $8,000. If the cost of accommodation, per diem and transportation are added, the total cost might be well-over the $21,700 per month which is the standard JSA cost for a long-term expert.

(3) Results of the Questionnaire Survey with Experts 153. In the questionnaire survey with experts, which is reported in Chapter 3 (please refer to ”3.3.2 Expert Survey” within section “3.3 Evaluation Results of Questionnaire Surveys” in this report), the experts responded to a question on the level of monthly rates.


38

When they were questioned about the major differences related to monthly rates for IMF IMF/JSA TA projects, more than half of the respondents (54.2%) answered that they believed the monthly rate of IMF was lower than other institutions, and the remaining (45.8%) agreed that the monthly rate for IMF was higher. The experts responded that the other organizations for which they worked before were the World Bank, ADB, UN Agencies, EBRD and IDB. It is assumed that a comparison was made with these institutions. From this result, the monthly rate paid to experts seems to be at a moderate level, or at least not particularly high.

3) Necessity of JSA and its Charging System for MOF as Well as IMF

154. JSA funds have been allocated to subject areas such as fiscal, financial, money market, and macroeconomic statistics. In these areas, demands for TA by recipient countries are expected to continue within the globalization trend of the financial market. Since JSA supports mainly low- to middle-income countries, the demand for improving macroeconomic and fiscal management as well as stabilizing the financial market is still high. Thus, JSA is expected to continue its support to these areas.

155. On the side of Japanese financial authority, given the tightening national budget situation, there is a growing demand to improve efficiency within budget execution. There is strong pressure to streamline and rationalize the use of funds in order to explain its value and appropriateness to the taxpayers. How should Japan respond to this trend and justify the contribution of JSA funds in this stern environment? As indicated in Chart 2.2 previously, annual contribution of JSA amounted to roughly around $20 million since 1996. In order for MOF to continue this support to JSA and fulfill its obligation to explain to the public, the introduction of a “charging system” seems to be an option to consider. The OTM report “Policy for Country Contributions for Capacity Building” (August 2008) explains the need for this scheme. The structure is shown in Chart 3.7.

Chart 3.7 Proposed Charging Regime for Non-program Countries (Percent of cost base)Income Group

- Low income : 10%

- Lower middle income : 30%

- Upper middle income : 50%

- High income : 100%

Note: The groups are based on GNI per capita, adjusted annually

(Source: Policy for Country Contributions for Capacity Building, August 2008 by OTM)

156. The charging system takes into account country income level. Currently, this scheme is applied only for internally financed IMF TA. Externally financed TA including JSA


39

is not currently subject to this scheme, however, it would be worthwhile to consider the introduction of a charging system to externally financed TA.

157. The Office of Budget and Planning (OBP) revealed in its plan of TA cost allocation for the future with a decrease in internally financed TA and an increase in externally financed TA. The next Chart 3.8 shows the projected budget breakdown as presented by OBP. 13

Chart 3.8 TA Cost Allocations, FY2008-FY2011(Millions of US$)

FY2008 Budget

FY2009 FY2010 FY2011

Direct cost of TA (real term based on 2008) 107.3 106.4 102.2-110.6 100.1-111.6

Externally financed 34.2 44.7 42.4-50.7 42.5-54.0

Internally financed 73.2 61.7 59.8 57.6

("The FY2009-FY2011 Medium-Term Administrative, Restructuring, and Capital Budgets," OBP.)

158. This allocation plan indicates that IMF TA has to depend on external financing more heavily. Moreover, according to the information we obtained from OTM, it is mandated by the Executive Board to substantially expand TA, and the main financing source will be through new and strengthened partnership programs with donors. Starting from a baseline of US$40 million annually, external financing is expected to be tripled in the medium term. In this context, the substantial effect of the charging system on IMF/JSA TA would be limited because this charging regime would only generate gross revenue of US4-6 million annually, which is only a small portion. The application of a charging system for low income countries, with limited financial resources, can be arranged by counting in-kind contribution recipient authorities, e.g. office space and administrative support, which is common in bilateral assistance. This issue needs to be discussed further.

13 This issue is discussed in the sub-chapter “V. Financing of TA” on page 26 in the OTM report, “Enhancing the Impact of Fund Technical Assistance, April 2008”


40

3.3 Evaluation Results of the Questionnaire Surveys

159. The purpose of this survey is to measure the achievements, effectiveness and impact of the IMF Technical Assistance and Training programs funded under the Japan Administered Account for Selected Fund Activities (JSA). The findings and lessons learned from this survey will be used for future improvement of IMF Technical Assistance and Training programs.

160. There were two audience groups specifically targeted for this survey. The first group targeted was beneficiaries who are counterparts of the recipient country government organizations. They are typically governors of central banks and general directors of ministries that received IMF TA financed by JSA. The second group targeted was experts, employed by IMF for JSA projects, who worked directly with the beneficiaries in a short-term or long-term assignment at the recipient government organizations in their respective countries.

161. The evaluation team identified a total of 201 targets for the survey. There were 124 requests sent to expert consultants and 77 requests sent to beneficiaries. The survey had an excellent response rate. We received 83 completed surveys (67% completion rate) from the expert consultants and 38 completed surveys from the beneficiaries (50% completion rate) in the project countries.

162. In addition to the multiple choice questions, some of the questionnaires, for both beneficiaries and experts, ask for narrative comments on specific issues, which have provided relevant information for the evaluation of JSA.

163. The questionnaire surveys were set up and compiled through “Survey Monkey”, an online company with tools to help design, host, and support our two surveys. Survey Monkey was chosen primarily because of their advanced tools allowing responses to be collected and analyzed multiple times and at multiple intervals. In addition to their support and analysis tools, Survey Monkey provided a secure environment for the surveys, requiring username and password protection to both the surveys themselves, as well as to the results.

3.3.1 Beneficiary Survey

164. For the beneficiary survey, there were 38 completed surveys from 12 countries. The Asia-Pacific region consisted of the following countries: Indonesia, Vietnam, the Philippines, Thailand, Cambodia, and Timor-Leste. Africa consisted of the following countries: Liberia and Egypt. Europe consisted of the following countries: Macedonia and Kosovo. South Asia consisted of the following countries: Afghanistan and Nepal.


41

1) Information related to the respondents and what type of organization

Organization: Answers for Question No.2165. A majority (70.7%) of the beneficiary respondents work with the Central Bank organization within their country. Other notable organizations the beneficiary respondents worked with include the Ministry of Finance (14.6%) and the Revenue and Customs Organization (12.2%).

Chart 3.9

Please check the types of organizations you work for (Multiple Selections Allowed)

0.0%

10.0%20.0%

30.0%40.0%

50.0%

60.0%70.0%

80.0%

Central Bank Ministry ofFinance

Revenuesand CustomsOrganization

FinancialSector

SupervisoryAuthority

NationalStatisticsAgency

Other (pleasespecify)

Specialized Area: Answers for Question No.3166. For the purposes of this survey and the beneficiaries, we focused on three main areas of expertise: Monetary Policy and Financial Systems, Fiscal Policy and Policy Management and Macroeconomic and Financial Statistics. A majority of the respondents worked within the Monetary Policy and Financial Systems area of expertise (57.6%), while the Fiscal Policy and Policy Management area of expertise had the second most number of respondents (33.3%) and the Macroeconomic and Financial Statistics had the fewest number of respondents. We also requested the respondents within each of these three groups to further break down their areas of expertise. These include:

Monetary Policy and Financial Systems (57.6% - 19 respondents), with 13 respondents (59.1%) in bank supervision and regulation.

Fiscal Policy and Policy Management (33.3% - 11 respondents), with 8 respondents (72.7%) in budgeting and public expenditure management.


42

Macroeconomic and Financial Statistics (18.2% - 6 respondents), with 7 respondents (77.8%) in monetary and financial statistics.

2) Evaluation Results

Topic 1) Relevance of IMF/JSA TA (Number of respondents: 39)

Consistency with government priority: Answers for Question No.4167. Regarding assistance provided by the IMF under the JSA in terms of consistency with government priorities, all of the responses were either excellent or good, and no answer for modest or poor.

Answer Response Share

Excellent 25 64.1%

Good 14 35.9%

Modest 0 0

Poor 0 0

No knowledge 0 0

Relevant comments to the assistance provided in the box below.

Egyptian Central Bank – I would like to underscore that there was an extensive preparation period before the launch of the actual TA program. This preparation period included extensive discussions with the IMF regarding the contents of the TA, the selection of experts, the chronology of material delivered and all other details. The experts arrived knowing exactly what was requested of them and there was a continuous dialogue between us (the authorities) and the IMF to ensure that the program work program was as designed.

Ministry of Finance, Indonesia – The assistance provided by IMF-JSA during the month of March 2005 was very timely since that signaled the start of the implementation of the new finance and treasury laws which were just passed by the Indonesian parliament. The March 2005 consultancy was the beginning of a series of IMF/JSA funded consultancies to support the Indonesian Central Government’s effort to reform its financial management system, specifically in the treasury area. The IMF Consultant was instrumental in the implementation of many reform initiatives in the Ministry of Finance, most particularly the implementation of the Treasury Single Account (TSA) and the drafting the government regulation on Cash Management.

Topic 2) Effectiveness of IMF/JSA TA Projects (Number of respondents: 39)

168. When discussing the outputs of IMF IMF/JSA TA projects, the responses again were similar, and fell between Excellent and Good.


43

Achievement of planned activities: Answers for Question No.6Answer Response Share

Excellent 16 41.0%

Good 23 59.0%

Modest 0 0

Poor 0 0

No knowledge 0 0

Use of TA outputs: Answers for Question No.7Answer Response Share

Excellent 19 48.7%

Good 18 46.2%

Modest 2 5.1%

Poor 0 0

No knowledge 0 0

Determinant factors for effectiveness: Answers for Question No.10169. Regarding the indication of determinant factors for IMF/JSA TA effectiveness in terms of experts’ capacities, the beneficiaries provided the following response:

Chart 3.10

Please describe the magnitude of roles played by expert(s) for IMF JSA-funded TA in assisting your

country. Select all that apply.

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%

Experience inspecific issue

Practical skills(analysis,solution)

Communication,coordination,and facilitation

skills

Leadership,motivation, and

personality

Managementand

administrationskills

Results for each of the Factors: 84.6% (33 respondents) - Experience in specific issue


44

76.9% (30 respondents) - Practical skills (analysis, solution) 66.7% (26 respondents) - Communication, coordination, and facilitation skills 35.9% (14 respondents) - Leadership, motivation and personality 28.2% (11 respondents) - Management and administration skills

Relevant comments regarding the roles of the experts include: Cambodia Central Bank - After JSA TA was provided, the progress has been made in strengthening the internal audit function within the bank by establishing the internal audit department, audit procedure manual and training new recruited auditors to enable them to conduct audit in professional manner. However, some areas such as auditing reserve, note printing, IT and bank supervision requires further assistance to guide our auditors.

Topic 3) Impacts of the IMF/JSA TA (Number of respondents: 35 to 39)

Internal Impact: Answers for Question No.11 (1)170. Regarding the internal impact of the IMF/JSA TA, the following responses were provided:

Chart 3.11

Please Indicate Internal Impacts of the IMF JSA-Funded TA

Improved Knowledge for

Reforms61.5%

Improved Knowledge and

Motivation for Reforms

51.3%

Improved Clarification on

Duties and Respsonsibilities of Government for

Reforms 20.5%

Improved Policies and Their

Instruments48.7%

61.5% (24 respondents) - Improved knowledge for reforms 51.3% (20 respondents) - Improved awareness and motivation for reforms


45

20.5% (8 respondents) - Improved clarification on duties and responsibilities of government for reforms

48.7% (19 respondents) - Improved policies and their instruments

Specific relevant comments include: Timor-Leste Central Bank - In-house capacity for policy design and implementation has been developed further as part of the preparatory work for the transformation of the Business Process Analysis into the Central Bank of Timor-Leste. Cambodia Central Bank - The new concept of internal audit and internal control has been known to all officers in the bank through the introduction of internal control policy which requires all departments to establish and maintain certain levels of control within their operations and dissemination of information about the roles and functions of internal audit. In addition, once recommendations were issued, several corrective changes have been made to improve the current operations to minimize the risks and ensure the reliable information or data, and the efficient use of assets.

External Impact: Answers for Question No.11 (2)171. Regarding the external impacts of the IMF/JSA TA, the following responses were provided:

Chart 3.12

Please indicate external impacts of the IMF JSA-funded TA.

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

Socio-economicimprovements

Increased awareness andknowledge of general public

Reputation among generalpublic for being recipient of

IMF JSA-TA

57.1% (20 respondents) - Socio-economic improvements 62.9% (22 respondents) - Increased awareness and knowledge of general public


46

8.6% (3 respondents) - Reputation among general public for being recipient of IMF/JSA TA

Specific comments include: Cambodia Central Bank - Recognizing the importance of internal audit functions, the National Bank of Cambodia requires all commercial banks and other financial institutions to establish internal audit unit to oversee their daily operations in order to ensure the soundness of banking system.

Topic 4) Sustainability of IMF/JSA TA (Number of respondents: 37 to 39)

Institutional memories: Answers for Question No.12 (1)172. In regard to rating the levels of sustainability of IMF/JSA TA, in terms of institutional memories within their organizations, the overall responses were “excellent” (13 respondents or 33.3%) and “good” (25 respondents or 64.1%), while only one respondent replied “modest” (partially satisfactory). There were no “poor” or “no knowledge responses.”

Capacity building of staff: Answers for Question No.12 (2)173. In regard to rating the levels of sustainability of IMF/JSA TA, in terms of capacity building of staff, a majority of the responses were “excellent” (15 respondents or 38.5%) and “good” (21 respondents or 53.8%). However, 3 respondents (7.7%) replied modest (partially satisfactory). There were no “poor” or “no knowledge” responses.

Relevant comments regarding the sustainability of IMF/JSA TA: Viet Nam Revenues and Customs Organization - In the progress of enhancing capability of the General Department of Taxation (GDT) in order to provide better service to the community, the GDT has embarked on several phases of tax reforms. The TA has contributed a lot of invaluable recommendations, consultancy to assist GDT to be obtained successful outcomes that are recognized by various stakeholders Cambodia Central Bank - Though the internal audit function has gradually moved forward, sometimes the project was terminated in the middle of the audit assignment which had been left uncompleted, or timing of the expert assignment is delayed due to the financial resource constraint.

Topic 5) Advantage of the IMF/JSA TA (Number of respondents: 37 to 39)

Answers for Question No.13174. Regarding whether the IMF/JSA TA have advantages/strengths compared to other bilateral and/or multilateral technical assistance in the fiscal, monetary and statistical areas, 17 respondents or 45.9% responded “Yes, very much so.”


47


Yes, very much so 17 45.9%

Yes, relatively so 10 27.0%

Not so much 1 2.7%

Not at all 0 0


Specific relevant comments include: Egyptian Central Bank - A key advantage of the IMF/JSA TA is that it provides enough flexibility for the authorities to design the TA based on the real needs but also ensures enough discipline since the TA is provided under the auspices of the IMF. I believe that for any TA program to be successful, the authorities need to play a leading role in following up and communicating with the IMF if there are any potential areas of confusion with the experts. Nepal Ministry of Finance - TA enhanced the implementation of other T.A. for the consolidation of ASYCUDA project funded by ADB/UNCTAD. Further, it has played a role to mobilize donor support for the reform program. Therefore, TA for the coordination of the reform program and mobilizing the donors for overall improvement in customs administration is felt necessary for future also.

Topic 6) Visibility of Japan as a Donor (Number of respondents: 39)

Recognition by Minister Level: Answers for Question No.14175. Regarding whether the IMF/JSA TA project is widely recognized by high ranking officials of the respondent’s government (such as ministers) as JAPAN-funded support, the response was as follows:

Chart 3.13

Is the IMF JSA-funded TA project widely recognized by high rankingofficials of your government (such as ministers) as JAPAN-funded

support?

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

Yes, very muchso.

Yes, relativelyso.

Not so much/ Not at all. No knowledge


48

Results from each response: 30.8% (12 respondents) - Yes, very much so 33.3% (13 respondents) - Yes, relatively so 12.8% (5 respondents) - Not so much 2.6% (1 respondent) - Not at all 20.5% (8 respondents) - No Knowledge

Contact with Embassy or JICA: Answers for Question No.17176. Regarding whether there has been any contact with the Japanese Embassy and/or Japanese aid implementing organizations such as JICA, in the implementation stage of IMF/JSA TA, most of the responses were “no” (30 respondents or 78.9%).


Yes 8 21.1%

No 30 78.9%

Overlap of JSA and JICA: Answers for Question No.18177. Regarding whether there has been any overlap between IMF/JSA TA and other technical assistance projects conducted by Japanese aid implementing organizations such as the Japan International Cooperation Agency (JICA), most of the responses were “not so much” (9 respondents or 23.1%) and “not at all” (20 respondents or 51.3%).


Yes very much so 0 0%

Yes, relatively so 1 2.6%

Not so much 9 23.1%

Not at all 20 51.3%


Specific relevant comments include: Indonesian Central Bank - Other TA’s do the same thing in bank supervision issues with a different focus on examination field. Viet Nam Revenues and Customs Organization - Generally, the General Department of Taxation (GDT) received TA from IMF and was not informed that these TA was financed by Japan so GDT was not aware of the TA provided by IMF was financed by Japan.

Conclusion of Beneficiary Survey Results

178. Overall, the beneficiaries seem to be content with the achievements, effectiveness and impact of the IMF Technical Assistance and Training programs funded under the JSA.


49

The role and performance of the experts were also evaluated as being helpful to the beneficiaries. The knowledge and awareness of JSA funded activities increased during the projects, however, visibility of Japan as a donor is not high, and there seemed to be little or no involvement or contact with the Japanese Embassy or the Japan International Cooperation Agency. This seems to be an area where improvement is necessary in the future.

3.3.2 Expert Survey

179. For the expert survey, there were 83 completed surveys from three main areas of expertise: Monetary and Capital Markets, Fiscal Affairs, and Statistics.

1) Information Related to the Respondents, Employment Status and Type of Contact with the IMF. Employment status: Answers for Question No.1 (1)180. IMF consultants on a contract basis with permanent job represented 88% - 73 respondents, contract base with no permanent job represented 10% - 8 respondents, and IMF staff represented 2% - 2 respondents. Long-term or short-term: Answers for Question No.1 (2)181. A majority of the experts’ assignment period consists of short term projects (60% - 51 respondents) versus long term projects (40% - 34 respondents), while there were two respondents answered as worked for both short and long term. TA Departments in Contact: Answers for Question No.1 (3)182. The following are the departments within the IMF that the experts are predominantly in contact with.

Chart 3.14

Which Department are you Mainly in Contact with at the IMF

Statistics Department (STA)

11.6%

Fiscal Affairs Department (FAD)

40.7%

Monetary and Capital Market

Department (MCM)48.8%


50

40.7% - Fiscal Affairs Department (FAD) 48.8% - Monetary and Capital Market Department (MCM) 11.6% - Statistics Department (SA)

Experience in Region: Answers for Question No.1 (4)183. The Regions where the experts primarily conducted IMF Technical Assistance projects:

48.9% - Africa 61.4% - Asia and Pacific 13.6% - Eastern Europe 20.5% - Middle East and Central Asia 9.1% - Latin America and Caribbean

Chart 3.15

In which region are you mainly conducting IMF technical assistance projects?

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

East Africa West Africa Other partsof Africa

Asia andPacific

EasternEurope

Middle Eastand Central

Asia

LatinAmerican

andCaribbean

Other International Organizations: Answers for Question No.1 (5)184. Regarding whether the experts had contracts with any other international organization other than the IMFJSA TA in the past 5 years, approximately half of the respondents (51.1%) replied “yes” while the remaining respondents (48.9%) replied “no.”

185. Regarding which organizations the experts surveyed had contracts with, the following responses were received:


51

Other organizations include: 56.3% - IMF (other than IMF/JSA TA)

26.6% - World Bank 10.9% - Asian Development Bank (ADB) 7.8% - United Nations (UN) Agencies 1.6% - European Bank for Reconstruction and Development (EBRD) 3.1% - Inter-American Development Bank (IDB) 21.9% - Not Applicable

Chart 3.16

At which organization do you have contracts with?

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

IMF (otherthan IMF JSA-

TA)

World Bank AsianDevelopmentBank (ADB)

United Nations(UN) agencies

European Bankfor

Reconstructionand

Development(EBRD)

Inter-AmericanDevelopmentBank (IDB)

Not Applicable

Difference Between IMF/JSA and Other Organizations: Answers for Questions No.1 (7)186 The questionnaire requested responses for the major difference between IMF/JSA and other international organizations, in terms of five aspects below.

Points of Difference Answered as “yes” The TOR is more detailed for IMF/JSA 77% Quality is higher for IMF/JSA 92% Monthly remuneration is higher for IMF/JSA 46% The contract is more complicated for IMF/JSA 31% The regional office is more supportive for IMF/JSA 87%

Relevant comments include: Expert for ArmeniaA) The IMF/STA JAS TA management system is the BEST known to me. This refers to all the

aspects, including: a) Appointment procedures, both administrative (no bureaucracy at all) and technical, both correspondence and briefing (if any) - (highly valuable and helpful support and guideline);


52

b) Backstopping during the assignment -very helpful, of the highest quality and always supportive; c) Administrative arrangements and HQ contacts: very supportive – all with no exclusion; d) Completing the assignment – same as above: very helpful always supportive, highest quality; e) Reporting: during the assignment (either monthly reports or quarterly – depending on the contractual arrangement): high quality requirements; limits in size, no limits in content; very useful. f) Final Reports and debriefing, if included: very well established system in terms of the quality requirements, highest quality review by back-stopper and at the HQ; publishing and sending to the authorities of the TA recipient country; g) Relationships with the host institution: highest ethical and professional requirements to the expert combined with the relatively independent position of the expert – all very supportive for the project to be implemented successfully. h) Liaising with the Resident Representative Offices and the IMF Country Missions: highest standards, always very supportive and helpful, both for advisor and the authorities, and vice-versa. (i) Transportation, security and immunity and other relevant arrangements: very satisfactory.

B) The last comment: Participation of the countries in charging future TA project, even partially, will most likely weakened the TA System, as it will reduce the independent position of the experts, create extra bureaucracy, increase reporting burden in expense of the work on the implementation of the project, possibly have also negative impact on security, immunity and other relevant arrangements with the authorities, and complicate other accepts as well.

2) Evaluation Results

Topic 1) Relevance of IMF/JSA TA (Number of respondents: 83)

Consistency with Government Policies: Answers for Question No.1187. Regarding the rating of assistance provided by the IMF/JSA TA in terms of consistency with recipient government priorities, a majority of the respondents (68.7%) chose excellent.


Excellent 57 68.7%

Good 23 27.7%

Modest 2 2.4%

Poor 0 0

No knowledge 1 1.2%


53

Overlap with Other TA: Answers for Question No.3188. Regarding whether the experts recognized any “overlap” between the IMF/JSA TA projects and other technical assistance projects provided by other multilateral/bilateral donors, 50 respondents (60.2%) replied there were no “overlaps” at all.


Yes 21 25.3%

No 50 60.2%


Additional comments include: IMF Staff, Statistics DepartmentThe objectives of the project are discussed in close consultation with recipient countries, which ensures the adequacy of actions undertaken with countries needs. There is strong cooperation with other TA providers in the same area (AFRISTAT, Commission of the West African Economic and Monetary Union, African Development Bank, French Statistical Office) National Accounts Advisor, Ministry of Finance, ArmeniaJust One example: the last assignment to Liberia (Feb 2009 - Feb 2010) / One year only: Liberia’s balance of payments statistics for 2004-2007 and metadata were produced for the 1st time since 1987 and published in Nov 2008, available online on the website of the CBL (http://www.cbl.org.lr/doc/major/bop2004-2007nov202008.pdf) : In the instruction says: "In recognition of the achievement, the CBL is expressing a special gratitude to the IMF and its Statistics Department for providing long-term technical assistance in balance of payments statistics to Liberia and the Government of Japan for financing the project. Also this gratitude extends to the Resident Balance of Payments Statistics Advisor to Liberia ...., whose support was of a great importance. This has engendered a new wave of confidence in the capacity of the CBL Research Department’s ability to collect, analyze, compile and produce standardized BOP and external trade statistics. Taking into consideration that this edition of the BOP statistics is compiled for the first time since 1987, the CBL’s Management would like to thank all data providers for the cooperation received during the process.” IMF Consultant, Asia & Pacific DepartmentIMF/JSA TA, particularly in banking supervision is at the core of financial sector issues - since strengthening regulation and supervision, is the key linchpin in promoting a stable financial system. In this way, IMF/JSA TA - in the banking supervision area-has an enormous degree of relevance and importance for not only broader financial system stability, but by extension, the well-being of the real economy. The recent global financial crisis reinforces the importance of maintaining a sound financial system through a robust regulatory and supervisory framework.


54

Topic 2) Effectiveness of IMF/JSA TA Projects (Number of respondents: 83)

189. Achievement of planned activities: Answers for Question No.5 Regarding the rating of IMF/JSA TA in terms of achieving the “planned activities and outputs”, a majority (85.6% - 71 respondents) stated that the achievements were excellent or good.


Excellent 39 47.0%

Good 32 38.6%

Modest 11 13.3%

Poor 0 0

No knowledge 1 1.2%

Answers for Question No.6190. Regarding the assessment on the “use” of IMF/JSA TA outputs, 52.4% (43 respondents) assessed it as “excellent” and 35.4% (29 respondents) as “good”.


Excellent 43 52.4%

Good 29 35.4%

Modest 7 8.5%

Poor 0 0

No knowledge 3 3.7%

Additional comments include: Expert, PhilippinesFor a program to be effective, combination of the appropriate type of assistance to the appropriate type of TA provider and support mechanism is needed. In my observation, the IMF, generally speaking, has been able to select good people and has been able to provide good backstop mechanisms. Expert, CanadaOn my assignment, negotiating the mandate (terms of reference) with the country was difficult as there was some difference of opinion on what the main focus of the TA should be. A compromise was reached and I consider that the TA provided was highly relevant and useful, and was very well received. Expert, Timor-LesteThe targets and outputs expected from a TA cannot be fully achieved within a short time frame of a year or two especially in a country where the capacities in Government are very low. Acceptance of new ideas for change takes a long time and putting them into practice takes an even longer time. Commitment for funding of technical assistance in countries where capacities are exceedingly low should be a long term period.


55

Topic 3) Sustainability of IMF/JSA TA

Capacity Building: Answers for Question No.8 (2)191. Regarding the rating of the levels of sustainability of IMF/JSA TA, in terms of internal capacity building of the counterpart staffs, a majority of the respondents (72.9% - 59 respondents) stated that the levels were either excellent or good.


Excellent 17 21.0%

Good 42 51.9%

Modest 15 18.5%

Poor 1 1.2%

No knowledge 6 7.4%

Additional comments include: IMF staff, Asia & Pacific DepartmentThe key to sustainability in my opinion is ensuring the receipts staff has a solid understanding of the concepts - this often requires significant training with references to examples and case studies. Generally, if staff is comfortable with a new concept it is more likely to be continued. Expert, UKJapan has made a remarkable contribution to much IMF TA in many countries of the World irrespective of Japan's spheres of immediate interest. Japan has not attempted to limit the recruitment of experts to its own nationals.

Topic 4) Visibility of Japan as a Donor (Number of Respondents: 83)

Awareness of Expert: Answers for Question No.9 & 10192. Regarding whether the experts are aware that the IMF/JSA TA projects are financially sponsored by the Japanese government, 96.3% - 79 respondents answered “yes.” Of all of the respondents who replied “yes”, they were questioned how they knew that the TA was being financed by Japan. The following are their responses.

57.1% - From IMF Staff 22.1% - From IMF TOR 18.2% - From IMF Notification Letter 2.6% - From other IMF Documents 0.0% - Not Applicable


56

Chart 3.17

Other responses include: IMF staff, Asia & Pacific DepartmentWhen funded by JSA, the TA contract will clearly and prominently indicate that the funding for the activity is being provided by a grant from the Japanese Government. Expert, Timor-LesteDuring the induction mission, IMF staff had specifically mentioned that the TA was being funded by the Government of Japan. The head of the Mission also made it a point to introduce me to officials in the Embassy of Japan in Dili so that the Embassy officials were aware of the TA funded by their Government.

Awareness by Recipients: Answers for Question No.11193. Regarding whether the recipient governments were aware that the IMF/JSA TA projects are financially sponsored by the Japanese Government, 83.6% - 61 respondents stated that they were, while only 16.4% - 12 respondents stated that they were not.

Activity to increase visibility: Answers for Question No.12194. Regarding whether any of the experts had observed or done any of the following activities related to the visibility of Japan as a donor in their projects:

94.6% (35 respondents) stated they had an oral explanation to participants at the beginning and/or ending of sessions

32.4% (12 respondents) stated they had a written explanation to participants in handout materials

16.2% (6 respondents) stated they had a visual presentation in handout materials, such as country name and/or national flag

10.8% (4 respondents) stated they were in attendance at the Japanese Embassy to sessions.

If the above answer is “Yes,” how did you know your TA is being financed by Japan?

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

From IMF Staff From IMF TOR From IMFNotification

Letter

From other IMFdocuments

Not Applicable


57

2.7% (1 respondent) stated they were in joint activities with the Japanese Embassy 2.7% (1 respondent) stated they had an explanation from the media

Chart 3.18

Have you Observed or Done any of the Following ActivitiesRelated with Visability of Japan as a Donor, in your Projects

05

10152025303540

OralExplanation toParticipants atthe Beginningand/or Endingof Sessions

WrittenExplanation toParticipants in

HandoutMaterials

VisualPresentation in

HandoutMaterials such

as CountryName and/orNational Flag

Attendance ofthe JapaneseEmbassy to

Sessions

Joint Activitieswith the

JapaneseEmbassy

Explanation toMedia

Ideas for Better Japanese Visibility: Answers for Question No.14

195. The comments by experts for better Japanese visibilities include: IMF Consultant, CanadaIn my opinion, the government of Japan and the people of Japan deserve visibility for their generous sponsorship. Some discretion is required, however, as in some cases, the recipient country may be embarrassed that they require such support. So I think the sponsorship should be visible, but that the visibility should be handled in a very tasteful and modest way, perhaps through the Japanese embassy's participation in an initial meeting with senior officials, for example. IMF Consultant, BrazilIt is valuable information for citizens in recipient countries, first of all. They value knowing which countries are acting in good will, and I believe that may be also valuable information for Japanese taxpayers: knowing that their money is being productively used in assisting needed nations. IMF Consultant, LiberiaThere is tremendous potential for really positive contributions to the developing countries in the world. There is no reason why the country/government providing the resources should not get recognized for the good work being done. How it should be done is another issue! One thought is that at the stage when TA is decided on, the donor country should be advised to the recipient


58

government and a joint meeting between the IMF, Recipient Country Government and Donor Country Government should discuss the requirement for recognition of the donor country and how to best achieve this.

Official Contact with Japan: Answers for Question No.15196. When the experts were surveyed if they had any official contact (such as coordination) with the Japanese Embassy and/or Japanese aid implementing organizations such as the Japanese International Cooperation Agency (JICA) in the implementation stage of IMF/JSA TA, 89.0% - 73 respondents responded that they had no such contact.

Conclusion of Expert Survey Results

197. Overall, the experts questioned in the survey seemed to be familiar with the IMF/JSA TA projects and believed that the effectiveness of the programs was above satisfactory. The experts also believed that the sustainability of the JSA TA programs was at a high level and would be useful in future projects. It also appears that both the experts and the individual target countries were aware of the funding by the JSA for specific projects. One notation is that there is very little contact with the Japanese Embassy and/or Japanese aid implementing organizations such as the Japanese International Cooperation Agency (JICA) related to JSA TA projects.

4. Evaluation Results through Field Survey

59

Chapter 4 Evaluation Results through Field Survey

4.1 Evaluation Results of Field Interviews

4.1.1 Cambodia

1) Overview of JSA Activities in Cambodia

198. Cambodia is one of the three countries14 among the APD countries to receive the largest TA resource allocation. According to the Regional Strategy Note (FY 2009-2011), Cambodia has received 11.74 person years (FY 2006-2008) and 10.10 person years (FY 2007-2009), which rank the first and second as a recipient country, respectively, out of a total of 33 APD countries. In fact, Cambodia was the largest recipient of TA among all countries in FY 2003, when the resource allocation was boosted under a multi-donor Technical Cooperation Assistance Program (TCAP)15. To date, the country has remained an intensive user of the IMF/JSA TA, and the JSA is its only funding source. The TA funded by the JSA in Cambodia covers comprehensive areas of reform; (i) Banking Supervision, (ii) Central Bank Operations/Audit, (iii) Tax and Customs Administration, (iv) Public Expenditure Management, and (v) Multi-sector Statistics.

Chart 4.1 IMF/JSA TA Projects in Cambodia Year: 2004-2008Sector Number of

Projects Commitment Amount Total

Note

Banking Supervision / Central Bank Operations (related with supervision issues)

7 $1,165,050 Long-term RAs, Short-term experts

Central Bank Audit / Central Bank Operations (related with internal audit issues)

5 $491,850 Good practice case Short-term experts, Peripatetic experts

Customs Administration / Tax and Customs Administration

5 $1,111,750 Good practice case Short-term experts, Peripatetic experts

Public Expenditure Management

4 $800,000 Long-term RAs, Short-term experts

Multi-sector Statistics 3 $611,900 Long-term RAs Total 21 $4,180,550

199. This trend, however, will be changed over the coming years with the aim to normalize the intensive resources that have been injected to Cambodia, in pursuit of its more efficient utilization. A significant decline is expected in the resource allocation, which will be narrowly focused on key macroeconomic constraints. Following the IEO evaluation 14 The other two countries are Indonesia and Timor-Leste. 15 The TCAP is a three-year medium-term technical corporation plan in support of the PRGF (1999-2003). It was formulated and agreed by the Cambodian government and all concerned development partners (including the Fund) in May 2000, and Launched in October 2003.


60

(2005) that concentrated on the analysis of the TCAP years (2001-2003), the amount of the TA allocation will be decreased.

2) Overall Evaluation

200. The evaluation team in Phnom Penh held interviews with recipients at the National Bank of Cambodia (NBC), General Department of Customs and Excises (GDCE), National Institute of Statistics (NIS), and Ministry of Economy and Finance (MEF), and with the World Bank, Asian Development Bank (ADB), and Japan International Cooperation Agency (JICA), from other developing partners. The results in brief are as follows:

(1) Effectiveness 201. The recipients consider the IMF/JSA TA quite effective in an overall sense. Based on the TOR, the experts have been providing TA, taking into consideration both country and global contexts in a pragmatic way. Ownership of the recipients seems to be strong. Several recipients pointed out that the length of the peripatetic experts was short in light of the recipients’ actual needs. The selection of experts by the IMF was appropriate, and the experts’ effort and attitude in assisting the counterparts was well-accepted. The fact that the same experts were repeatedly requested also proves the recipients’ approval of the quality of their services.

(2) Impact 202. IMF TA has produced tangible impacts such as establishing an internal audit system at the NBC, and introducing Customs Law and related regulations, National Accounts, and the Producers Price Index (PPI). Legal framework and regulations such as Customs Law, Statistical Law and related Decrees have been prepared with the assistance from IMF TA and other donors’ assistance, aligned with a forward-looking view to Cambodia’s WTO accession16.

(3) Visibility of Japan 203. Japan’s visibility could be verified within the higher managerial level at the recipient organizations; however, front line officers do not necessarily recognize it. Visibility could mainly be enhanced in two ways: 1. increasing symbolic exposure, 2. providing substantial contribution in collaboration with Japan’s other assistance schemes, such as creating synergies with JICA projects. (Coordination with other donors aiming for synergy and complementarily would enhance effectiveness of IMF TA, in general.)

(4) Good Practice Cases 204. Two good practice cases were selected in the areas of Central Bank Audit (NBC) and Customs Administration.

16 Cambodia became a member of WTO in October 2004.


61

3) Evaluation by Major Sectors

205. Based on the interview surveys in Phnom Penh and reviews of relevant documents, the following are the highlights of the findings with the premise of these two factors: Given the broadness of the IMF/JSA TA and on-going assistance of other development partners in the relevant areas, progress may reflect the joint efforts of the IMF and others – hence, attribution is unclear. In addition, because the reform process consists of continued efforts on the part of the Cambodian authorities and development partners, “marginal” progress is not always easy to measure in the objective form.

(1) Banking Supervision206. NBC has been receiving long-term resident advisors (RAs) on banking supervision since 1996 on and ongoing basis. All successive RAs have come from the same organization– the Bank of France – and have been contributing to the NBC’s massive needs to keep up with the rapid growth of banks. RAs have been serving as effective resources for issues related to banking supervision in a flexible manner. Short-term experts, assisting to strengthen on-site and off-site supervision have been complementing RAs’ work in strengthening the NBC organization and auditing.

207. The level of TA and the use of long-term RAs are expected to remain stable taking into account of the recipient’s reform needs, priority and efficient utilization of input resources. With respect to further strengthening the payment system, the World Bank and the ADB are expected to provide primary support.

208. During the interview, the Cambodian authorities gave high praise for the IMF/JSA TA to the NBC in providing extremely useful support on banking sector reform issues. They also expressed their sincere appreciation of Japan’s contribution as a sponsor – Japan’s visibility seems to be strong. They expressed that hands-on assistance from long-term RAs is a “must”, and considered as a “best practice”, emphasizing that the IMF has been the only provider of such high quality assistance. The authorities mentioned that the IMF/JSA TA has been in line with the Financial Sector Blueprint for 2001-2010 as well as the Financial Sector Development Strategy 2006-2015.

209. The authorities consider the choice of the TA modality – long-term RAs – appropriate to meet their needs. The authorities consider short-term experts, assisting to strengthen on-site and off-site supervision, just as important as RAs. However, they mentioned that the length of assistance may be a little short in light of the increasing complexity in the banking system.

210. Other major development partners providing assistance to the NBC are the World Bank, ADB and the IFC. All necessary information on TAs would be consolidated to the RAs – hence RAs are in a position to advise the NBC with comprehensive perspectives to help avoid overlaps and create synergistic effects among different TA's.

211. The Cambodian authorities referred to the following outputs and outcomes as RAs’


62

work in banking supervision. Successive RAs have provided support for: 1) drafting the Law on Organization of NBC (1996), Foreign Exchange Law (1998), and Banking Law (1999), 2) implementing banking system restructuring, 3) setting up micro finance institutions, 4) recapitalizing banks for bail out, and 5) issuing regulation on commercial banks.

(2) Central Bank Operations/Audit (Banking Inspection)212. As mentioned above, the Fund’s TA to the NBC to strengthen internal audit would be considered as a good practice case, and this part will be elaborated in section 4.2 Cases of Good Practices.

(3) Tax and Customs Administration213. By the same token as above, this part will be discussed in section 4.2 Cases of Good Practices.

(4) Public Expenditure Management 214. The Fund’s TA in the area of public expenditure management originates from the multi donor program, TCAP, which included the IMF, the World Bank, ADB, the UNDP, and DFID. FAD has supported the MEF on the public expenditure management components of the TCAP in the form of long-term RAs. In 2004, the Prime Minister approved a comprehensive public financial management (PFM) reform program (PEMRP)17 under the SWAPS18 framework, and the IMF has continued to play an important part. In fact, the TA resulted in implementation of a new chart of accounts, as from the 2007 annual budget.

215. TA in this area will decline slowly, in light of a better management of resources – the modality of the Fund’s TA has changed to a peripatetic-based input from 2006. As many development partners are involved in this area including JICA, coordination among relevant donors becomes a crucial factor to achieve expected results of the TA.

216. In the interview, the Cambodian authorities acknowledged that the experts that the Fund dispatches have been the best in terms of quality among all the experts sent by development partners to the country. As for the modality of TA, they would prefer long-term RAs to the current form of peripatetic experts, in light of the country’s vast needs for appropriate fund administration that would link to the broader picture of the reform on revenue and expenditure management. 19

17 The World Bank works as a focal point for coordination on development partners’ side.18 Sector Wide Approach (SWAP's) is the TCAP’s successor framework for “functional” coordination platform. SWAP's was introduced to provide a more holistic and better coordinated approach for assistance. In fact, the TCAP’s capacity bottlenecks have revealed its lack of substantive coordination among developing partners and have become an obstacle to the Fund’s TA implementation.19 According to FAD, there are different views within the MEF. (The views of the Director General of the National Treasury are not reflected here as he was not present in the interview. Most of FAD TA has been directed toward the National Treasury (NT), and, according to FAD, there are challenges in providing TA to the NT, which is amongst the weakest MEF departments in terms of capacity and drive for results. It would have been an important input if the evaluation team could hear the views from the Director General of NT in this respect.)


63

(5) Multi-sector Statistics217. The IMF (STA) has been providing assistance in statistics prerequisites, national accounts, balance of payments, government finance, and monetary and consumer price statistics since 2001. Training has been given to compilers to strengthen their statistics capacity as well.

218. The main recipient is the National Institute of Statistics (NIS), Ministry of Planning, MEF, and NBC. Coordination among relevant ministries and organizations on the recipient side has been taking place with the utilization of the General Data Dissemination Standards (GDDS). According to the Regional Strategy Note, in line with the transition from the broad-based emphasis of the resident Multi-Sector advisor to more targeted short-term inputs focused on capacity building, assistance in the form of long-term RAs ended in 2008. The NIS expects continued support through short-term advisors. Other development partners including JICA, SIDA (Sweden), UNDP, and UNFPA have been active in this area, and close coordination is required to avoid overlaps and inefficiency.

219. In the field interview, the Cambodian authorities emphasized the importance of the IMF/JSA TA to the country’s reform process in strengthening statistical capacity. They mentioned that the reform process has been undertaken very successfully, with assistance from Multisector Statistics Advisors in statistics prerequisites, national accounts, balance of payments, government finance, and monetary and consumer price statistics, which are the very core of the reform. The JSA experts kept good relations with the NIS staff and provided adequate advice, with an understanding of the Cambodian context. Country ownership has been secured in the sense that the final decision would be made by the Cambodian authorities themselves, regardless of the inputs and advice the experts would suggest. 220. The authorities mentioned that capacity building of the NIS staff has been successful and they have acquired the necessary skills to collect, analyze and disseminate statistical data on their own. The recent switch of the TA modality from long-term RA to peripatetic form has not affected their daily operation substantially. The NIS has been able to produce annual reports, quarterly forecasts on GDP data, price statistics etc. without assistance from experts. The number of staff members has increased from 130 in 1998 to 305 in 2009, which indicates one aspect of the reinforcement of the country’s statistical system.

4) Visibility of Japan

221. Japan’s high visibility among the NBC staff may be a reflection of the recently conducted IMF/MOF (Japan) field visit to the NBC under the JSA in January, 2009. As for other organizations, only staffs at the managerial level seem to recognize Japan’s contribution to the Fund’s TA. Nonetheless, all the authorities interviewed showed understanding and support towards Japan’s position to enhance visibility because of the importance of the IMF/JSA TA to the country’s reform process. One organization expressed their willingness to help strengthen Japan’s visibility within their organization – by


64

utilizing the opportunity of a monthly meeting summoning chief managers in the region, they would appeal to Japan’s contribution to the IMF/JSA TA.

5) Conclusion

222. The authorities showed appreciation for the effective assistance provided by the IMF/JSA TA. The authorities esteemed the quality of the JSA experts’ work and relevant contribution to support the country’s reform process. Generally speaking, the IMF/JSA TA has been aligned with the Cambodian strategy and ownership is respected. On the other hand, several authorities pointed out that difference in opinion exists on the sequencing and the pace of reform, although basic consensus has been made on the overall direction of the reform on both sides. As for the modality, some authorities wished for long-term RAs in term of the growing need for capacity building and strengthening institutions. Coordination among development partners seems to be carried out well.

4.1.2 Indonesia

1) Overview of JSA Activities in Indonesia

223. Among the 33 countries which are covered under the operations by APD, Indonesia is among the top recipients of TA resource allocation. According to the Regional Strategy Note, Indonesia has received 11.33 person years (FY 2006-2008), which was the second largest after Cambodia, and 14.42 person years (FY 2007-2009), which was the first country in the ranking. In comparison with other South East Asian countries such as the Philippines, Thailand, and China, it is notable that in FY2009, Indonesia received funds with a sharp increase to facilitate the implementation of fiscal and financial sector reforms.

224. The TA is a significant component of IMF engagement with Indonesia supporting the Indonesian Government’s reform agenda. The country has remained an active user of the IMF/JSA TA, and the JSA seems to be a dominant funding source. The TA funded by the JSA in Indonesia covers areas of reform such as; (i) Banking Supervision and Regulations, (ii) Public Financial Management/Treasury Management, (iii) Market Risk Supervision, (iv) Central Bank Operations and (v) Anti-Money Laundering (AML) and Combating the Financing of Terrorism (CFT).

Chart 4.2 IMF/JSA TA Projects in Indonesia Year: 2004-2008Sector Number of


Note

Banking Supervision and Regulations

5 $740,700 Good practice case Long-term RA, Short-term expert

Public Financial Management/Treasury Management

5 $521,100 Good practice case Short-term peripatetic expert


65

Market Risk Supervision 1 $179,400 Short-term peripatetic expert

Central Bank Operations 1 $155,100 Short-term experts Anti-Money Laundering (AML) and Combating the Financing of Terrorism (CFT)

1 $330,000 Short-term experts

Total 13 $1,926,300

225. According to an RSN observation of the recent macroeconomic environment, low public sector debt ratios have resulted from policy implementation and a favorable external environment. In addition, the corporate and banking financial situation has generally been sound. Economic growth has been steady, while inflation has been relatively contained. Large increases in international reserves facilitated Indonesia’s early repayment of remaining obligations to IMF in October 2006. Investment, including FDI, is still low and the financial sector remains small compared to other Asian countries. Meanwhile, the RSN points out that the economy is still vulnerable to investors’ sentiment change. The authorities aim at keeping a sound macroeconomic policy framework while implementing a broad structural reform agenda. In order to sustain growth, investments in infrastructure should be further promoted, and reform efforts should still be made to strengthen economic institutions, improve macroeconomic management, reduce corruption, and promote transparency and good governance. The TA Priorities are set based upon the above observations.


226. The evaluation team conducted interviews with government officials at Bank Indonesia (BI) and the Ministry of Finance (MOF), both of which are major recipients. Also, interviews were held with an IMF Resident Representative, one IMF Expert who has been serving as a long-term advisor for bank supervision at BI under JSA TA, and with the World Bank and Japan International Cooperation Agency (JICA).

(1) Effectiveness 227. According to the recipients, the IMF/JSA TA (wherein experts have been sent to Bank Indonesia and the Ministry of Finance), is considered very effective and fruitful. In both cases, the subjects covered are specialized areas, i.e. bank supervision and treasury cash management. The advice given by the experts was found to be effective, backed by sufficient expertise and practical experience accumulated in their former careers. The selection of experts by IMF headquarters was deemed appropriate, and the experts’ efforts and attitude in assisting their counterparts were well accepted by the Indonesian side. The fact that the same experts were repeatedly requested also proves the recipients’ approval of the quality of their services.

(2) Impact 228. IMF TA has a major advisory role in the area of bank supervision and policy level issues at Bank Indonesia. The presence of IMF is highly dominant, compared to other


66

donors which have a very limited role in TA for policy areas at BI. On the other hand, in the public financial management (PFM) area with Ministry of Finance as the counterpart, the “PFM Multi Donor Trust Fund” which is organized by five donors’ funding20, covers technical assistance for institutional reform for PFM. This Trust Fund comprehensively covers the area of PFM restructuring, and in that overall picture, IMF/JSA TA is focusing on the treasury function; especially for cash management or establishing a Treasury Single Account (TSA). According to DG Treasury, the realization of a treasury single account has a comparatively large impact and benefit for the treasury, compared with the limited inputs.

(3) Visibility of Japan 229. In both cases of BI and MOF, recipients were not aware, prior this evaluation visit, that this IMF TA was financed by Japanese contributions through JSA. JICA, a Japanese bilateral ODA agency, seems not to be informed of the existence of experts sent by JSA. The need for exchanging views with Japanese experts attached to MOF should be considered for possible future exchange between IMF/JSA and JICA experts.

(4) Good Practice Cases 230. The two cases, bank supervision and treasury advisors, were selected as good practice cases. Both of the cases have been well accepted and are regarded as successful TA by the Indonesian counterparts.


231. The following are highlights of the findings, which are based on interviews in Jakarta and on reviews of relevant documents including the Regional Strategy Note.

(1) Banking Supervision and Regulations 232. Extensive support to BI in banking supervision and regulation was provided as one of the top priority areas. BI received a long-term resident advisor (RA) on banking supervision starting in 2004. The key areas of advice include risk-based supervision, consolidated supervision, and movement toward Basel II implementation. Also, as a related support, a peripatetic expert in risk-based modeling (for Basel II) is also being provided.

233. The current RA assigned to BI is a former U.S. regulator in the banking area, who had been working for the Federal Deposit Insurance Corporation (FDIC) in California. He subsequently took assignment as an expert in the central banks of Bulgaria and Thailand. The assignment with Bank of Thailand was also sponsored by the IMF/JSA. During the interviews, the Indonesian authorities expressed high regard for the IMF/JSA TA to BI in providing high quality, practical and useful support on strengthening bank supervision capacity.

234. Regarding the mode of TA, the BI executives emphasized the importance of having long-term RAs always around them. The authorities consider it very important that the

20 EU, the Netherlands, Switzerland, Australia and Japan (PHRD)


67

advisor is always available to BI staff for advice and consultation, whenever issues or questions arise.

235. Regarding donor coordination in the area of banking supervision and regulations, BI has no other major assistance than IMF, which occupies a dominant position within BI.21

(2) Public Financial Management/Treasury Management 236. Achieving treasury reforms and fiscal transparency are also priority areas. The Ministry of Finance set the objectives to implement a fully operational Treasury Single Account (TSA) and improving cash forecasting and planning. IMF TA has supported these efforts through the sending of a peripatetic advisor. There have been significant developments in the area of management of government accounts and cash balances, including the conclusion of an MOU between the Ministry of Finance (MOF) and Bank Indonesia (BI) aimed at more efficient cash management.

237. Regarding donor coordination, RSN points out that the joint mission of IMF and the World Bank, which was sent in December 2008, on asset and liability management provided a number of recommendations, including strengthening the coordination between cash and debt management. Thus, for IMF, to guide the Indonesian Government in overall budget reform strategy, jointly with the World Bank, remains a priority agenda. It is said that this budget reform strategy possibly includes medium-term expenditure framework (MTEF) and program budgeting reforms, as well as budget execution.

(3) Market Risk Supervision and Central Bank Operations 238. The assistance in market risk has been provided to BI in the form of peripatetic experts to build capacity for market risk supervision. TA for the central bank operations was provided to help reorganizing BI branch offices.

(4) Anti-Money Laundering (AML) and Combating the Financing of Terrorism (CFT) 239. IMF has also been supporting AML/CFT efforts, with further involvement expected through the implementation of an externally financed multi-year program on legal drafting and capacity building of supervisory institutions in various sectors, including remittances.

(5) Other Areas 240. In addition to the above assistance, according to the latest RSN available, the following areas are described.

241. Fiscal: Tax administration remains a key priority, both to raise collections and to make the system more business friendly. The IMF has assisted the authorities in formulating a strategy, as well as providing a resident advisor and peripatetic visits by experts, including most recently in the areas of high-wealth individuals and collection enforcement. Another area of focus is improvement of Reports on the Observance of Standards and Code (ROSC),

21 There are other donors present in BI such as GTZ of Germany. However, GTZ provides technical assistance in the area of support to SME's and microfinance.


68

particularly through better oversight of public enterprises and sub-national governments, strengthening of audit procedures, and public disclosure of fiscal risks.

242. Statistics: Priority is given to assistance in improving fiscal data, including the reporting of government finance statistics (GFS) and sub-national governments’ accounts following the decentralization process. Also important is improved national accounts data, including ways to eliminate the statistical discrepancy to better measure export and import deflators.


243. The two major recipients of IMF/JSA assistance, MOF and BI were not aware that the IMF TA is financed by the Japanese contribution through JSA. Both of the institutions were informed of the fact just before the visit by the evaluation team for this study. The RA in banking supervision recognized the funding source at the beginning of his assignment, since it was indicated in the employment agreement with IMF. Unfortunately there was no particular occasion for the RA to interact or exchange views with Japanese stakeholders, such as other Japanese experts or government officials. As for an idea of possibly sharing his work experiences and expertise with Japanese ODA stakeholders in the future, the RA was found to be very positive and keen to do so in the future.

244. There is little exchange of information between IMF experts and JICA. In order to improve Japanese visibility, several possible countermeasures were discussed during the interviews. The authorities and individuals interviewed showed their understanding and support toward Japan’s position to enhance its visibility as a donor, mainly due to the fact that IMF/JSA has been playing a very significant role. The IMF Resident office also confirmed that JSA and Japanese visibility should be further improved and there might be several ways to accomplish this in the future.

5) Conclusion

245. It was found that IMF TA, although its funding source is not recognized, is considered to be of high quality and successful by the Indonesian counterparts. They praised the expertise and hands-on advisory services provided by the experts. One of the officials pointed out that the quality of IMF TA is maintained not only by the expertise of the experts themselves, but also by the quality control provided by the IMF headquarters, where the staffs at functional divisions also generally have sufficient knowledge and experience. The backstop function of headquarters is highly appreciated.

246. As far as the two major areas (bank supervision and treasury reform) are concerned, the IMF/JSA TA is well targeted and consistent with the Indonesian priority. The ownership of the Indonesian Government has also been high and respected. On the other hand, as pointed out before, there has been no recognition of the funding source, and there is a need to enhance Japanese visibility by joint efforts among IMF and Indonesian counterpart agencies.


69

4.1.3 Vietnam

1) Overview of JSA Activities in Vietnam

247. Among the APD’s diverse set of member countries, Vietnam is considered one of the emerging market countries that is to receive a fair size of the IMF TA. According to the Regional Strategy Note (FY 2009-2011), Vietnam received 3.35 person years between FY 2007-2009, which ranks as the 10th out of the total of 33 APD countries. In fact, Vietnam saw a sharp increase in TA allocation in FY 2009, especially on fiscal and financial sector issues. The major areas of the TA funded by the IMF/JSA in Vietnam are: (i) Tax Administration, (ii) Banking Supervision and Regulation, and (iii) Monetary Policy and Operations as listed below. Multi-donor Trust Fund (MDTF) managed by the World Bank also provides funding to the Fund’s TA activities. A number of development partners have been providing TA in these areas, and the IMF TA has a high tendency to focus its assistance on the overarching policy level.

Chart 4.3 IMF/JSA TA Projects in Vietnam Year: 2004-2008Sector Number of


Note

Tax Administration 6 $872,200 Good practice case Peripatetic experts

Banking Supervision and Regulation*

1 $77,550 Peripatetic expert

Monetary Policy and Operations

1 $232,650 Peripatetic expert

Total 8 $1,182,400 * The long-term RA in this area has been requested in the Project Proposal for FY 2009.


248. The evaluation team in Hanoi (August 24-26, 2009) held interviews with recipients at the General Department of Taxation (GDT) and the State Bank of Vietnam (SBV) – Banking Supervision Department and Monetary Policy Department. The team also interviewed the World Bank staff, and experts and a staff member from the Japan International Cooperation Agency (JICA) as well as from other development partners, and the Fund’s Senior Resident Representative in Vietnam. The following are the main results:

(1) Effectiveness 249. The recipients interviewed mentioned that IMF/JSA TA is quite effective, while its magnitude and coverage have been relatively focused on Vietnam. The recipients have utilized the advice of experts in a selective way, which can be considered as an indication of the Vietnamese side’s strong ownership. Regarding the TA modality, the officials of the State Bank of Vietnam were quite appreciative of the firm contribution provided by a resident advisor on banking supervision. Other Vietnamese authorities appreciate short-term and


70

peripatetic experts in light of effective utilization of resources. The selection of experts by the IMF was appropriate, and the experts’ effort and attitude in assisting the counterparts is well-accepted.

(2) Impact 250. IMF TA has contributed in producing tangible impacts in modernizing tax administration by creating the GDT’s strategic plan; by rolling out self-assessment reforms to all GDT tax offices; by realigning GDT’s organizational structures; by strengthening banking supervision; and, by identifying a roadmap on banking supervision system reform, including off-site and on-site inspection, assisting the revision of Decision 493 (on loan classification and provisioning) and Decision 457 (on prudential ratios). The IMF/JSA TA is aligned with Vietnam’s overall objectives aimed at establishing a market economy.

251. It is noteworthy that several interviewees mentioned the fact that external factors have contributed to facilitate the reform process. To put it concretely, Vietnam’s efforts toward WTO accession22 with strong political will and leadership, and direct and indirect pressure to deal with various issues emerging after its accession are said to have served as a tailwind for making progress. The authorities find urgent need for capacity building to put up with and overcome the changing environment of the financial arena. In the face of mounting challenges as economic integration deepens, the role of IMF and its TA would become increasingly important.

(3) Visibility of Japan 252. Japan’s visibility seems to be limited to a higher managerial level and to those who are involved in the TA in their daily operations. Visibility could be enhanced mainly in two ways: 1) increasing symbolic exposure, and 2) providing substantial contributions in collaboration with Japan’s other assistance scheme such as creating synergistic effects with JICA projects.

(4) Good Practice Cases 253. Good practice cases can be taken up in the area of Tax Administration (GDT). Further information is required to describe the case, through IMF and/or direct contacts with the respective experts.

3) Evaluation by Major Sectors 254. Based on the interview surveys in Hanoi and reviews of relevant documents, following are the highlights of the findings based on two factors: 1) Given the broadness of the IMF/JSA TA and on-going assistance of the World Bank and other development partners in the relevant areas, progress may reflect the joint efforts of the IMF and others – hence, attribution is unclear. 2) Because the reform process is a continuous effort on the part of the Vietnamese authorities and development partners, “marginal” progress is not always easy to measure in the objective form.

22 Vietnam became a member of WTO in January 2007.


71

(1) Tax Administration 255. As mentioned above, the IMF/JSA TA to the GDT to promote voluntary compliance and increase revenue by improving taxpayer services and enforcement is considered as a good practice case, and this part will be elaborated in section 4.2 Cases of Good Practices.

(2) Banking Supervision and Regulation 256. In the areas of banking supervision and regulation, and monetary policy operations, the SBV has received peripatetic advisors. The relationship between the IMF and the SBV seem to be somewhat remote since the expiration of the IMF’s three-year loan arrangement with Vietnam under the PRGF on April 200423. Both parties could not reach agreement on measures by the Vietnamese authorities to meet the IMF general policy in regard to audit and accounting arrangements for the SBV.

257. However, in the face of mounting challenges to strengthen banking regulation and supervision, the SBV has shown increasing willingness for the IMF/JSA TA, and coupled with proactive outreach on the part of the IMF Senior Resident Representative, the IMF has been able to deploy a long-term resident advisor (RA) on banking supervision within the SBV organization in 2008. In light of the strong and widely-acknowledged ownership of the reform process by the Vietnamese authorities, no other development partner has come into play in a pivotal decision making process at the policy level. In this regard, the IMF considers this as a significant step forward to relationship-building with the SBV.

258. The Vietnamese authorities praised the quality of the long-term RA’s work, which commenced in 2008, mentioning that no other development partner can provide comparable assistance. The RA has had extensive hands-on experience and assistance to the economies in transition (former republics of the Soviet Union). The authorities stated their high appreciation of the RA’s contribution; especially on the improvement of on-site supervision and preparation of the strategy/roadmap on banking supervision system reform. The fact that the RA has been well accepted by the authorities seems to indicate the RA’s ability to grasp the current Vietnamese context, including institutional nuances, and to respond flexibly to their needs in the midst of the volatile fiscal situation in which the SBV is entangled.

259. The authorities admitted that some overlap of assistance among development partners existed in the past, which has now been solved. A JICA expert in charge of banking supervision noted that there is no overlap in their assistance and expects synergy with the RA’s work. CIDA (Canadian International Development Agency) has provided assistance mainly aimed at supporting banking supervision reform until 2020. One of the RA’s TOR is the coordination of assistance among development partners. As such, the RA is in the position to comprehensively view and conduct mapping of different assistance, and advise SBV for effective assistance that meet their real needs.

260. The authorities referred to the following outputs as the RA’s work so far in banking

23 Disbursements under the PRGF have been held up since late-2002.


72

supervision. They noted that some of the work is outside the scope of the RA’s, but are aligned to the SBV’s strategies and needs, and expressed deep appreciation for his flexibility. The outputs include:

Revision of the banking supervision system and preparation of its road-map (including both on-site and off-site supervision);

Revision of Decision 493 (on loan classification and provisioning) and Decision 457 (on prudential ratios);

Development of samples for the data reporting of banks to the SBV; Building financial soundness indicators, and Providing training program to the SBV staff

261. The authorities made a strong call for continued support from the Fund, and anticipated support through a combination of on-going long-term RA with peripatetic advisors to be in place (expected) to complement the RA’s efforts.

(3) Monetary Policy and Operations 262. The Vietnamese authorities mentioned that useful TA on monetary operations has been provided since 2005 in the areas of liquidity forecasting and core inflation measures. Amid the volatile financial market situation, the authorities, in close coordination with peripatetic advisors, have adopted the most suitable solutions for each occasion. The Fund is the only TA provider among core inflation measures. They highly value the advisors’ ability to apply what has been thoroughly considered at the policy framework level into the actual situation in Vietnam.

263. The authorities have also made use of international workshops arranged by the Fund to learn about concrete cases in other countries (the Philippines, Thailand, and Indonesia). They are quite satisfied with the advisor’s pragmatic advice on the ground. The World Bank and the Asian Development Bank (ADB) have been providing on-going assistance in the monetary operations but they both focus on specific issues (e.g., money laundering, payment systems) and thus, there is no overlap.

264. The authorities, confronted with a rapidly changing money market environment, consider peripatetic assistance an appropriate modality especially in this area. From the viewpoint of efficiency, they prefer intermittent support from advisors when new issues arise and need external assistance to overcome the problem.

265. The authorities referred to the following as the outcome: 1) Open market operations have now become daily operations; 2) the interbank market has become more active; and 3) closer coordination and increased information flow between the SBV and the MOF are taking place for enhanced liquidity forecasting.

266. The authorities emphasized that reform takes time, and flexible and prudent monetary policy should be taken up that addresses the actual conditions in Vietnam. Given the immature market situation they are facing, they foresee a long path to comply with


73

international standards. The management issues of the interbank market constitute an example of the situation.

267. On the other hand, the authorities acknowledged that the IMF stance has changed over time – the IMF has become more flexible to respond to their changing needs and has deepened understanding towards country-specific issues. They expressed the urgent necessity for training SBV staff to tackle mounting challenges, and they anticipated the IMF support.


268. As mentioned above, Japan’s visibility seems to be limited to a certain segment. The authorities are more concerned with the quality of TA, and many interviewees maintain their view that as long as they receive effective assistance, the funding source gains less attention. A suggestion was raised which would call on the IMF for information disclosure on their TA sponsors. In Vietnam, all ministries and agencies are required by regulation to report to the Ministry of Planning and Investment (MPI) on the contents (including funding sources, periods and man-months) of external assistance they receive every year. The authorities pointed out that if the IMF provides such information, visibility would be enhanced for all the sponsors, including Japan.

5) Conclusion

269. The authorities expressed overall satisfaction to the Fund’s TA and praised the outstanding work provided by the experts. They consider the modality of the TA appropriate and expect further assistance. Coordination among development partners has been carefully taking place to avoid overlap and to create synergy.

270. In Vietnam, the IMF/JSA TA seems to be covering a few selected areas of reform, in comparison with those of Cambodia, where the Fund has profound strength. Some interviewees pointed out that Vietnam’s strong national ownership for the reform process and the country’s solid bases for decision making and coordination may have affected the scale and the coverage of the IMF TA. Several authorities pointed out the changes in the IMF stance in providing assistance – towards a more country-oriented, pragmatic approach, and developing closer relationships.

4.1.4 Tanzania (East AFRITAC)

1) Overview of JSA Activities in the Country

271. During 2004 to 2008, there were 4 JSA projects approved and implemented including one project in Tanzania for banking legislation, two projects for East AFRITAC (in Tanzania) in PFM advisor and one project for Ethiopia and East African countries for customs. Besides these projects IMF/JSA financed 11 projects for East African countries mainly in the area of bank supervision and customs administration. Among these, the


74

field visit covered one project for Tanzania (Central Bank and banking legislation), two projects for East AFRITAC (PFM long term expert) and one project for Ethiopia and East African countries (Custom administration) which had contacts with East AFRITAC.

Chart 4.4 IMF/JSA Projects in Tanzania and East AFRITAC (2004-2008)Sector Number of

Project Commitment Amount Total

Note

Legal 1 $51,700 Good Practice Case Tanzania: Central Bank and banking legislation

Fiscal Affairs 2 $438,900 Good Practice Case

East AFRITAC: Public Finance Management Advisor,

1 $155,100 Ethiopia and East Africa: Customs administration

Total 4 $645,700

Chart 4.5 IMF/JSA Projects in other East African Countries (2004-2008)Sector Number of

Project Commitment Amount Total

Note

Monetary and Capital Market

8 $1,472,450 Rwanda(5) and Uganda(3): Central bank supervision

Fiscal Affairs 2 $257,400 Kenya: Customs administration, Uganda: Tax administration

Legal 1 $25,850 Kenya: Financial legislation Total 11 $1,755,700

272. According to the Regional Strategy Note (RSN) for the African Department (2008-2010), the subjects considered as strategic priorities for the African region are public resource management, financial sector development, and trade liberalization. In the fiscal area, TA to post-conflict and fragile countries are to receive priorities through fiscal advisory service by a resident advisor at East AFRITAC. For Tanzania, financial sector reforms continue to be the main focus of IMF TA, with priorities on deepening domestic financial market, monetary operation, foreign exchange operations and building capacity for financial stability reporting. In the area of PFM, TA will continue to be channeled through East AFRITAC and will focus on strengthening the institutional arrangements for PFM at MOF.

2) Overall Evaluation 273. The following are evaluation results based on interviews conducted by the evaluation team during August 24 to 26, 2009, in Dar es Salaam. The evaluation team had meetings with East AFRITAC, Bank of Tanzania, Ministry of Finance, and other relevant donors, The World Bank and JICA, Center Coordinator and resident advisors at East AFRITAC (experts in bank supervision and PFM) and IMF resident representative for Tanzania.


75

(1) Effectiveness 274. Through interviews with beneficiaries, BOT and MOF, the TA activities taken by East AFRITAC have been recognized as very effective, mainly due to regional establishment and quality of experts and practical contents of TA being delivered. The location of East AFRITAC, in particular for Tanzania, is advantageous in establishing communication with experts for various consultations, similar to home doctors in proximity. The presence of resident advisors familiar with the regional environment makes the difference compared with the mission type experts sent from home countries by other donors.

(2) Impact 275. The IMF/JSA TA on legislative issues for Bank of Tanzania (BOT), in particular, has a comparatively strong impact to the input (two short-term experts for one month of field time), since the TA was involved in reviewing the central bank act and banking regulation. For this TA, IMF recruited an ex-central banker from Sweden and a lawyer from the US. The formation of an international team enabled the Tanzanian authority to be assisted with international best practices for renewing the legal framework of the Tanzania financial sector.

(3) Efficiency 276. For East AFRITAC, the office space is located in a new building of the Bank of Tanzania, and support staff is also provided by BOT24. While it is being counted as a contribution to East AFRITAC by the Tanzanian Government as the host country, the office location and facility offers an efficient environment for experts and staff of East AFRITAC, considering frequent movements of long and short-term experts visiting neighboring countries to provide TA services.

(4) Sustainability 277. The physical presence of East AFRITAC in Tanzania and its achievements in delivering TA activities to member countries since its establishment in 200225, brings more expectations of the locally based TA Center under IMF. The evaluation report released in March 2009 reported that East AFRITAC is taking a leading role among the three AFRITAC’s (East, West and Central), and the overall evaluation results for East AFRITAC is scored as 3.1' which is the highest among the three AFRITAC's and positioned at the high end of successful (good) rating bracket. This indicates the sustainability of East AFRITAC as a project as well as TA activities delivered by regional establishment of IMF.

24 East AFRITAC moved to the current location in November 2007.25 Independent External Evaluation for AFRITAC’s by Bruce Murray and Team, March 2009, IMF


76

(5) Good Practice Cases 278. The following two projects are identified as the good practice cases to be described in detail for the JSA evaluation report.

East AFRITAC: long-term advisor for PFM (2006/2007) Tanzania: Central Bank and Banking Legislation (2005)


(1) Monetary and Finance Sector 279. IMF TA has a major advisory role in the area of bank supervision and monetary policy areas at the Bank of Tanzania, and its presence is dominant, compared to other donors which have no involvement in TA for policy areas in BOT. The JSA financed two short-term experts for Banking Act reform in 2005, to BOT, were evaluated as highly effective, and the Banking Act reform was realized based on these IMF/JSA experts’ contribution. BOT indicated its interest in receiving IMF/JSA TA, in case the subject is Tanzania-specific rather than a regional common issue, which can be covered by East AFRITAC.

280. Due to its proximity, BOT is a frequent user of resident advisor of East AFRITAC. The recent issues of BOT’s particular interest are bank supervision and financial market development, and a new issue is Islamic finance regulation which is relevant to some of the countries in the region. According to the interview, BOT Deputy Governor comments IMF TA through East AFRITAC as highly relevant and practical.

(2) Fiscal Affairs Sector 281. On the other hand, in the area of public financial management with the Ministry of Finance, the World Bank and other GBS support donors are forming a PFM Development Partners’ Group to conduct PFM Reform Programs in Tanzania. The role of IMF TA, among the donor’s group, can be for up-stream policy level advice and for highly specialized areas such as legal frameworks, cash management, fiscal reporting, budget classification and program budgeting. These are areas of high competence with the IMF.

282. According to the World Bank staff, which is taking a leading role in the PFM Development Partner Group26, East AFRITAC is contributing to some specific areas such as cash management and budget and public accounting systems, in which IMF’s technical contribution is highly effective. The issue of consolidating government bank accounts is also important. It is more a technical, rather than policy matter; however, it is considered to be the effective area where the IMF can mobilize its experience.

283. Japan is one of 14 countries participating in the Government Budget Support Group27 in Tanzania, and JICA is participating as an active member by allocating one staff

26 The group consists of 13 donors including Japan.27 AfDB, Canada, Denmark, EC, Finland, Ireland, Japan, Germany, Netherlands, Norway, Sweden,


77

member in charge of PFM at the JICA Tanzania Office and sending one expert for fiscal decentralization.


284. Since East AFRITAC is run by a basket-funded contribution by 16 donors28, it is not practical or possible to identify any particular activity as financed by a particular donor. The Japanese contribution through JSA financing for long-term experts is being considered as a parallel contribution to the basket fund, and recognized as one of 16 donors. The name of the donors, including Japan, appear on every document created and distributed by East AFRITAC and on its website29.

285. In case of IMF/JSA short-term experts sent to BOT for Banking Act reform in 2005, the BOT side was not aware of the fund source being from Japan. But now BOT realizes the IMF TA with JSA finance and indicated interest in utilizing JSA in case an issue specific to Tanzania should arise.

286. As the subject for future collaboration with JICA, simplification of customs procedures at border crossings in the East African region could offer a potentially interesting subject by both parties, since JICA is currently studying One Stop Border Post (OSBP) in Kenya for the purpose of smoothing traffic to reduce transportation costs, along with physical infrastructure development. The field survey interview with JICA indicated that the JICA field office has very limited information about IMF TA activities, including East AFRITAC, and they are not able to judge whether there is a common issue of interest. There seems to be room for improvement in terms of information flow with other donors.

287. There has been some contact established between the Japanese Government or ODA agencies and East AFRITAC. For example, a JICA mission visited East AFRITAC in November 2007, and a MOF representative participated in an East AFRITAC Steering Committee in March 2009.

Switzerland, UK and the World Bank28 AfDB, Canada, China, Denmark, Finland, France, Germany, Italy, Japan, Luxemburg, Norway, Russia, Sweden, Switzerland, Netherland and UK29 WWW.Eastafritac.org


78

Chart 4.6

Country $'000s ShareUK $3,455 15%AfDB $3,000 13%Switzerland $2,410 11%Netherlands $2,172 10%Italy $1,645 7%Japan $1,579 7%Other 10 countries $8,558 38%Total 1 $22,818 100%

1 Total excludes remaining funds from Phase I rolled-over into Phase II

Contribution to East/West AFRITAC by country, Phase II(May 2006 - October 2009)

Source: OTM

288. As one of 16 donors to East AFRITAC, “Japan” appears on any relevant documents of East AFRITAC, in the same manner as the other 15 donors. Chart 4.6 indicates the contribution amount share by major donors by percentage for the three year period from 2006 to 2009, for both East and West AFRITAC combined. The largest contributor is the UK, followed by AfDB, Switzerland, Italy and Norway. Japan represents a contribution at the modest level, though its contribution through JSA is only for East AFRITAC.

5) Conclusion

289. Although the number of JSA TA projects was limited to Tanzania and East AFRITAC, the projects observed were successful and evaluated as being highly effective. There are two projects to be selected as good practice cases which are elaborated in the next section. However, the visibility of Japan as a donor was not high in the case of the TA project for the Bank of Tanzania. In the case of East AFRITAC, Japan is positioned as one of the 16 donors. There seems to be a future possibility in JSA TA involvement, in the form of direct TA to the Tanzanian Government or through East AFRITAC for regional TAs

290. While there is no duplication observed among Japanese bilateral technical assistance and IMF/JSA TA, there is no active communication established between JICA and East AFRITAC regarding the issues in the area of PFM, which is being coordinated by the World Bank for the interested donors group. The possibility of achieving mutual benefit by establishment of communication is to be considered further.

291. As for regional interest, customs administration support by East AFRITAC might be of relevance with developing On Stop Border Post (OSBP) for border control of merchandise and products within the region. JICA is conducting a study in Kenya and neighboring countries in developing OSBP. The future possibility for collaboration regarding this issue will be an interesting topic.


79

4.2 Cases of Good Practices

292. In order to illustrate projects with effective results, cases of good practice were selected based on interviews with the recipients during the field visits. There are seven projects listed below with their descriptions in detail in this section.

Chart 4.7 List of Good Practice Cases Project Title Country

1 Central Bank Audit / Central Bank Operations (related with internal audit issues)

Cambodia

2 Customs Administration / Tax and Customs Administration Cambodia 3 Banking Supervision Advisor Indonesia 4 Treasury Advisor Indonesia 5 Tax Administration Vietnam 6 Central Bank and Banking Legislation Tanzania 7 Public Financial Advisor at East AFRITAC East Africa Region

4.2.1 Cambodia ----------------------------------------------------------------------------------------------------------------------- Case of Good Practice – Cambodia (1)Fiscal Years (Costs): 2008($59,800), 2007($114,400), 2006($162,600), 2005($77,550), 2005 ($77,500) IMF TA Department: MCM Project Titles: Central Bank Audit / Central Bank Operations (related with internal audit issues) Local Counterpart: National Bank of Cambodia (NBC), Internal Audit Department -----------------------------------------------------------------------------------------------------------------------

1) Background and Country Context

293. Peripatetic advisors have been providing assistance to the NBC in the areas of banking inspection since 2005. As a part of the bigger scope to strengthen the central bank operations, new functions were introduced in the NBC to enhance internal control and internal audit, following the diagnostic study (safeguard assessment) conducted to identify vulnerabilities and inefficiencies. Until then, the MCM has been working with the NBC for a number of years in enhancing its operations focusing on the banking supervision aspect. However, the results of the assessment highlighted the NBC’s significant weakness in the areas of internal control and internal audit, and establishing and building NBC capacity in these areas have become pressing issues to tackle. Consequently, an Internal Audit Department was newly created in the NBC and the IMF as provided by JSA TAs so that the Department would function effectively on an independent basis.

294. The JSA TAs would be useful in bringing the NBC more in line with international


80

standards in these areas. Now, the Internal Audit Department has 13 staff in place (1 Inspector General, 2 Deputy Inspector Generals and 10 auditors), which accounts for only about 1% of the total NBC staff. The authorities recognize the critical role they play, serving as the backbone to develop “good governance” in the financial sector (increasing transparency, reducing risks etc.) The authorities envisage further strengthening of their capacity and an increase in the number of staff to carry out its audit operations. Reflecting its strength and comparative advantage, the Fund has been the exclusive provider of assistance in these areas – no other development partner has come into play for assistance.

2) TA Activities and Outputs

295. The successive experts provided assistance in the following areas where an urgent need was identified to strengthen banking inspection capacity: 1) providing intensive on-the-job training to the internal audit staff in the conduct of internal audits, from the planning stage through to writing the report; 2) assisting the internal audit staff in designing audit programs to cover various activities in the NBC, including: (i) branch operations, (ii) the vault operations in the Issue Department, (iii) soiled notes activity in the Issue Department, and (iv) foreign exchange operations; and 3) assisting the internal audit staff identify weakness in the control arrangement, and in making appropriate recommendations for strengthening the framework.

296. As the most noteworthy tangible result of the Fund’s TA, the authorities stated that the Internal Audit Department is now undertaking independent internal audits of the 14 branches (out of a total of 24). Cash and banking operations have been independently audited in accordance with professional standards as well – hence detailed audit reports have been prepared for all the audits undertaken, and have been issued to the governor and other senior management responsible. On the other hand, the authorities noted that more complex areas of foreign exchange management, IT and note printing operations have as yet not been audited, and pointed out the necessity for developing audit procedures manuals and staff to be trained in these areas. To this end, the authorities expressed their intention to increase the number of staff and further strengthen the Internal Audit Department. While pending issues still remain, results seem to be coming out step-by-step and the authorities expressed their intention to receive further TA from the Fund.

3) Results and Comments from Recipients

(1) Comments on the overall evaluation of the IMF/JSA TA 297. The Cambodian authorities expressed appreciation to the high quality assistance provided by a peripatetic advisor on establishing and strengthening banking inspection capacity of the NBC since 2005. Because the banking inspection was totally a new function to the NBC, the staffs’ capacity building has always been a pressing issue, and the IMF support has become an integral part of the reform. The authorities recognize that the IMF TA is aligned with the Financial Sector Development Strategy 2006-2015 appropriately, and country ownership has been respected and strengthened through the TA process. As for the responsiveness of the IMF, the authorities pointed out that experts have not


81

necessarily been dispatched in a timely manner as an urgent necessity. They also expressed their wish that the duration of the advisors stay in Cambodia would be longer in order to cover the entire TOR – they consider the length of the peripatetic advisor’s stay too short in light of growing capacity building need.

(2) Comments on the Qualification of Experts 298. The authorities consider the following preeminent characteristics necessary for the qualified experts: rich work experience with deep technical knowledge in Asian countries including Cambodia, strong understanding of the “Cambodian context” and respecting ownership on the Cambodian side, and flexibly responding to emerging needs and the changing situation. A sustainable relationship of mutual trust between the authorities and experts backed by the above-mentioned characteristics seems to be the key. The authorities expressed satisfaction with the work of the expert – and the same expert has been assigned to do the work. The authorities also mentioned that the current expert is well accepted by the NBC staff, and provides effective assistance and training staff. As a result, the authorities would like the same expert to continue assisting them in the future.

4) Japan’s Visibility

299. The authorities were aware that the funding source of the IMF/JSA TA was from Japan, and they’ve expressed their appreciation to Japan for the financial contribution.

----------------------------------------------------------------------------------------------------------------------- Case of Good Practice – Cambodia (2)Fiscal Years (Costs): 2008($239,200), 2007($143,000), 2006($216,800), 2005($387,750), 2004 ($125,000) IMF TA Department: FAD Project Titles: Customs Administration / Tax and Customs Administration Local Counterpart: General Department of Customs and Excises (GDCE) -----------------------------------------------------------------------------------------------------------------------


300. The fiscal component of a multi-donor TCAP covers tax policy, tax administration, customs administration, and budget management (including treasury). The TCAP envisaged forward looking perspectives to achieve the WTO accession benefit. In support of the TCAP goals and implementation of expected reforms, RAs, short-term experts and headquarters-led missions (FAD) were dispatched to the GDCE in the areas of tax and customs administration prior to 2003. IT experts, in the form of peripatetic advisors, provided input to facilitate modernization through enhancing the IT system.

301. As indicated in the Project Assessment Reports, the TA in recent years has focused on selected key objectives in the 2003-2008 Customs Modernization and Reform Plan, including: 1) preparing regulations and customs instructions to implement the Law on


82

Customs; 2) pre-project preparation for implementation of the Asycuda computer system30;3) establishing the risk management office to implement the government’s border risk management policy; 4) implementing the WTO valuation rules to assist Cambodia to meet its obligations for WTO membership; 5) developing a new strategic plan for the period 2009-2013; 6) developing an information technology strategy; and 7) strengthening the national trade statistics system, pending full implementation of the Asycuda system in 2009-2010.

302. The modality of assistance has changed from long-term RAs to peripatetic short-term experts since 2003 for more efficient utilization of resources – hence, a gradual downward trend is expected. According to the RSN, the decline reflects the large investment and progress made in recent years and the increasing activity of other development partners including the World Bank in this area. JICA has also been providing assistance in this area in the form of specific projects.31


303. In light of the above background, successive experts have covered the following areas of strategic importance, which are consistent with the Customs Modernization and Reform Plan, 2003-2008: 1) Establishing an adequate legal framework (including drafting a revised customs law and its related regulations); 2) developing an effective strategic planning process; 3) phased implementation of procedural and organizational changes (including Asycuda implementation, changes in headquarter functions and delegations to provincial offices); 4) providing training to gain adequate technical capacity (especially on valuation, audit, classification and risk assessment); 5) drafting an IT development strategy; and 6) developing effective risk management in coordination with all border agencies.

304. Complementary TA's exist from other developing partners including the World Bank (funding UNCTAD’s Asycuda project) and JICA. The French customs administration has been focusing its assistance on the training of staff.


(1) Outcomes 305. As one of the biggest contributions of the IMF/JSA TA among others, the authorities made reference to the promulgation of the Law on Customs, which was passed by the National Assembly in 2007. In the preparation process, a special committee was set up within the government, and the experts have been heavily involved in assisting the 30 The World Bank’s Private Sector Development and Trade Facilitation Project is funding UNCTAD’s installation of the Asycuda World computer system in customs, together with trade facilitation reforms inother government agencies. However, the Project does not provide TA to implementation and thus, the authorities have requested the IMF’s support in this area.31 “Regional Cooperation Project on Risk Management for Customs in Mekong Region” (Feb.2008-Mar.2011) and “Capacity Building Project for the Tax Department of Cambodia, Phase 1(Jan.2004-Jan.2007) and Phase 2(Jun.2007-Jun.2010)”.


83

drafting process. Currently, with the IMF/JSA TA, all necessary regulations and instructions have been amended responding to the changes of the Law. Other results and future challenges include on-going efforts to developing IT strategy, preparation of action plans to implement the 2009-2013 strategic reform and modernization plan, designing measures for planning, monitoring and evaluating customs’ operational performance, and designing a new organizational structure and transition strategy.

(2) Comments on the Overall Evaluation of the Fund’s TA 306. The Cambodian authorities expressed their satisfaction towards the content and the modality of the IMF/JSA TA. The IMF/JSA TA focuses on the overall policy level where as other development partners provide assistance for more technical issues at the individual project level. The GDCE staff and successive experts have built up trustful relations, which they consider as a necessary condition for successful TA. The authorities attach importance to “continuity” – they would prefer receiving assistance from the same experts repeatedly, once the relationship of mutual trust has been established, provided that the experts have proven good performance in terms of both technical and practical knowledge on the ground. The authorities are content to the modality of the TA – because the issues and concrete actions are clear, getting assistance on a peripatetic basis as needs arise will suffice.

(3) Comments on the Qualification of Experts 307. The Cambodian authorities emphasized that the first priority, above all things, when they consider the qualification of experts would be their quality of work – not only in terms of theoretical knowledge but also practical experience on the ground and ability to integrate with the GDCE staff. As for assistance in drafting laws and regulations (such as the Law on Customs and related regulations), they think European experts have a comparative advantage for the familiarity of the legal system they handle.


308. The authorities had knowledge about the funding source of the Fund’s TA, which was coming from JSA, and they’ve expressed their gratitude to Japan for the financial contribution. In addition, the authorities admitted that they find something congenial with JICA experts – with common cultural nuances and the Asian way of thinking. Such being the case, they added that if qualification as an IMF expert is equally high (nationality being the only difference), then they would prefer Japanese experts, which offers suggestion to the visibility consideration.


84

4.2.2 Indonesia ---------------------------------------------------------------------------------------------------------------------- Case of Good Practice – Indonesia (1)Fiscal Years (Costs): 2008($260,400), 2008($89,700), 2006($81,300), 2004($228,000) IMF TA Department: MCM Project Titles: Banking Supervision Advisor Local Counterpart: Bank Indonesia (BI), Deputy Governor and Bank Supervision Department -----------------------------------------------------------------------------------------------------------------------


309. In Indonesia, the banking sector was at crisis in the midst of the Asian financial crisis in 1997 to 1998. Since then, the authorities have sought to strengthen the banking sector and its regulation and have made considerable progress. An important element of this progress has been the implementation of banking regulations and supervisory practices that are closer to international standards, notably the Basel Core Principles on Effective Banking Supervision. Nevertheless, important reforms are still needed and given the importance of Indonesia's economy in the region and the close linkages between economic, financial, and political stability in Indonesia itself, further technical assistance to strengthen banking regulation and supervision remains a high priority.


310. IMF experts have been placed with Bank Indonesia (BI) to advise on various areas of banking regulation and supervision. The IMF/JSA Expert at BI, who formerly worked for the U.S. Federal Deposit Insurance Corporation (FDIC), was expected to advise on a broad range of banking regulatory and supervisory issues, as agreed with BI. A particular emphasis was placed on: (a) risk-based supervision, (b) Basel II issues, (c) aligning supervision with the Basel Core Principles (BCP), and (d) further developing stress testing at BI.

311. As an overall objective, the TA was aimed at assisting BI in improving banking regulation and supervision. More specifically, to strengthen BI methodologies and practices in key areas of bank supervision and regulation, and in preparing for the implementation of Basel II. At the conclusion of the expert's assignment, it is expected that measurable progress will have been made in the areas identified above.


312. According to the interviews with recipients, the expert provided very useful and significant policy advice to strengthen BI methodologies and practices in key areas faced by BI. Overall, clear progress was made toward implementing risk-based supervision, in Basel Core Principle (BCP) compliance and preparing Basel II, which helped strengthen key areas of bank supervision and regulation in Bank Indonesia. The specific TOR Items which


85

have been covered by the expert are as follows: Advise the authorities on the implementation of risk-based supervision Advise the authorities on the implementation of Basel II Advise the authorities on its compliance with the 25 Basel Core principles for effective

banking supervision. Provide ad-hoc advice as needed on the development of prudential regulations that are

issued to the banking industry. Train supervisory staff on a range of topics Other Initiatives required by BI, such as provision of ad-hoc advice to BI’s Board of

Governors on a range of regulatory and supervisory policy issues and delivery of numerous high-level presentations to BI’s Board of Governors and the Directors of banking policy and supervision.

313. Regarding coordination with other donors, there has been no other donor assistance in the area of central banking policies and operations. IMF has kept its dominant position in BI. AusAID has been providing TA to the Ministry of Finance on the governance and privatization of the state-owned banks, but never overlapped with the IMF TA in BI.

314. According to the interviews conducted by the evaluation team, executives and middle managers at BI, who have been closely working with the Expert, have a very high regard for the work and performance of the expert. They were satisfied with the expert’s high-level technical skills and expertise, the way of communication, and practical and hands-on advice. The Deputy Governor of BI implied that, for the future, he would like to keep the same expert continuously within BI with an expanded role and responsibilities, such as more computerization for banking supervision, establishment of a training center with research functions, etc.

315. From the interview survey, several key factors for success were identified. First, the expert had broad and deep prior experience as a banking regulator in the U.S. FDIC. In addition, before assignment in Indonesia, he had been working as an advisor in central banks in Bulgaria and Thailand, which gave him abundant experience on how to provide customized, practical advice in the cross-cultural context. The authorities commented that the expert’s advice is always well thought out and fits well with the Indonesian environment. The second factor is his personality and communication skills with BI staff, being well accepted by Indonesian counterparts for his well rounded and amicable character.


316. The funding source of JSA TA was not recognized by the BI side. The expert had knowledge about it at the start of the project since it was described in his employment agreement. However, there were no particular events or efforts to publicize the funding source during the TA activities. The expert is very supportive of ideas, for the future, to promote JSA such as visiting Japan to share his experiences with the Japanese ODA community.


86

---------------------------------------------------------------------------------------------------------------------- Case of Good Practice – Indonesia (2)Fiscal Years (Costs): 2008($119,600), 2006($81,300), 2006($216,800), 2005($51,700) IMF TA Department: FAD Project Titles: Treasury Advisor Local Counterpart: Ministry of Finance, DG Treasury -----------------------------------------------------------------------------------------------------------------------


317. After Indonesia completed an extended arrangement with the IMF in December 2003, post-program monitoring was done by IMF staff to maintain a policy dialogue with the Indonesian authorities. During 2002-2004, the authorities took positive action including the adoption of new laws for the budget and the treasury, and a restructuring of the Ministry of Finance. In 2004, the then Minister of Finance requested TA for assisting the Directorate General of Treasury to rationalize government banking arrangements and improve cash management capacity, and to eventually establish a Treasury Single Account (TSA). To that end, the Fiscal Affairs Department (FAD) of IMF provided a peripatetic advisor to the Directorate General of Treasury. Priorities for the Directorate General of Treasury (DG Treasury) include: (i) improved budget execution procedures; (ii) rationalization of government banking arrangements; (iii) consolidation of off-budget funds; (iv) establishment of effective cash management capacity, including a Treasury Single Account (TSA); and (v) strengthened debt management. Over the longer term, a move toward performance budgeting and accrual accounting is planned.


318. The advisor was expected to provide assistance in reforms in government banking arrangements in the DG Treasury. The approach was made in two stages: first, rationalize the bank accounts that are already under the control of the DG Treasury, including the 178 regional treasury offices (“KPPN's”); and second, begin to integrate into the TSA the bank accounts that have, for a long period of time, been outside the control of the DG Treasury. The overall objective of the TA was to rationalize government banking arrangements in Indonesia, particularly through establishing effective cash management capacity, including a Treasury Single Account (TSA). As a result, the project was expected to facilitate improved budget execution procedures, provide rationalization of government bank accounts, provide integration of extra-budgetary funds and treasury operations, and provide simplification of the payment and revenue remittance system, as well as improved cash planning integrated with debt management.

319. The major activities conducted under the TA were as follows; Improving budget execution procedures Rationalizing bank accounts Zero-balancing government accounts Introducing a new legal basis for government banking arrangements


87

Absorbing bank liquidity Forward cash planning Strategic advice on treasury computerization Streamlining treasury field offices

320. Regarding donor coordination, the IMF TA is closely related to the World Bank’s project to put in place an integrated financial management information system (hereinafter called “SPAN”, the acronym in Indonesian), which is assisting DG Treasury in its computerization of the budget and treasury accounting system. The SPAN is supposed to provide an umbrella structure for reforms in public financial management. IMF advisors' work plays a significant and complementary role for the success of DG Treasury’s computerization efforts, which includes the establishment of a TSA as a milestone. In the Ministry of Finance, The World Bank, along with AusAID, also provides TA in debt management, developing an overall debt management strategy, and providing TA in developing operations in securities.


321. Interviews conducted by the Evaluation Team revealed that the counterpart officials at DG Treasury, Ministry of Finance, who have been closely working with the expert, are highly satisfied with the expertise and performance of the expert. They feel that the IMF offers high quality TA, supported by the quality control function of the headquarters. The expert demonstrated excellent technical knowledge and skills, good communication and negotiation ability.

322. Prior to the assignment in Indonesia, the expert had been working for the Ministry of Finance in India and acquired sufficient expertise and experiences to satisfy the Indonesian counterparts. DG Treasury pointed out that, not only did the expert have deep knowledge, the negotiation ability demonstrated by the expert was excellent, and it played an important role in establishing necessary MOU's between MOF and other agencies. Under the umbrella of SPAN, the IMF TA focused on very specific areas such as rationalizing government bank accounts and zero-balancing government accounts. The expert was able to provide very useful advice in these special fields.


323. According to DG Treasury, the counterpart officials were not aware that the funding source of the IMF TA was JSA, namely Japanese contribution.

4.2.3 Vietnam ----------------------------------------------------------------------------------------------------------------------- Case of Good Practice – VietnamFiscal Years (Costs): 2008($239,200), 2007($114,400), 2006($108,400), 2005($206,800), 2005($103,400), 2004($100,000) IMF TA Department: FAD


88

Project Title: Tax Administration Local Counterpart: General Department of Taxation (GDT) -----------------------------------------------------------------------------------------------------------------------


324. Of the three major areas of the JSA TA, the Vietnamese authorities have been receiving substantial assistance in the areas of tax policy and revenue administration in the form of peripatetic advisors and headquarters’ missions (FAD) for support and technical backstopping. As indicated in the Project Assessment Reports, in the past, the TA efforts in this area have been considered less effective – exemplified by the fact that the authorities introduced VAT in 1999 that deviated significantly from the IMF recommendations.

325. However, their approach has changed and has brought in a cautious approach to reform with the gradual introduction of modern techniques and procedures by building-up firm results through pilot programs. By 2005, the authorities have come to fully agree with the IMF recommendations and have requested its further assistance. As pointed out by the GDT staffs and the IMF Senior Resident Representative, such a background depicts the importance of trust building and mutual understanding among the relevant stakeholders through a learning process on the basis of a pragmatic approach to undertaking reforms.


326. The GDT has received substantial TA in support of revenue administration: 1) testing new forms of tax administration at a small number of pilot offices and 2) formulating a five-year strategic plan aimed at modernizing the entire GDT by 2010. The FAD mission in June 2006 found that the pilot reforms had achieved positive results and the strategic plan was fundamentally sound. To support the national extension of the reforms, the FAD mission provided extensive advice on the following initiatives 1) reorganizing the entire GDT; 2) developing new strategies for administering small, medium, and large taxpayers (including the creation of a Large Taxpayer Office (LTO)); 3) strengthening the management of the GDT; and 4) improving the draft tax administration law. The peripatetic advisors provide assistance in preparing concept papers, action plans and timetables that help the GDT to implement the above-mentioned reforms.


(1) Outcomes 327. The authorities referred to the following outcomes in modernizing the tax administration that would be strongly linked with the contribution from the Fund’s TA: 1) The enactment of a new Personal Income Tax (PIT) law entering into effect on January, 2009; 2) preparation of the GDT’s strategic plan (2005-2010); 3) pilot implementation of modern self-assessment procedures based on international standards in order to facilitate


89

nation-wide rollout32; 4) reorganization of the GDT structure along functional lines reflecting the core tax administration functions; 5) decisions on the administration of small, medium, and large taxpayers; 6) groundwork to establish an LTO and agreement by concerned authorities for administrative restructuring, and 7) establishing a working group for the preparation of the GDT’s successive strategic plan up to 2020. Interviewees on the development partner side mentioned that external factors related to WTO accession and compliance with its requirements have, by and large, facilitated the progress.

(2) Comments on the Overall Evaluation of the Fund’s TA 328. In the interview, the Vietnamese authorities expressed their appreciation to the IMF/JSA TA with high quality advice based on its cross-country, comprehensive perspectives (including dissemination of good practices and lessons learned). They acknowledged that the capability to mobilize qualified experts with hands-on experience, profound technical knowledge, and the ability to work closely with the authorities through building on trusting relationship is the IMF’s strength. They consider the choice of the TA modality – peripatetic advisors – appropriate for the efficient use of resources. They make full use of e-mails to consult with the advisors when they are not in Hanoi, and seem quite satisfied with getting necessary advice through remote access.

(3) Remarks on the Approach to Reform 329. The Vietnamese authorities also pointed out the importance of a pragmatic approach to the reform process. They basically agree with the IMF on the overall direction of reform on tax administration modernization, and at the same time, they pointed out the importance of a pragmatic approach on the pace and the sequence of reform. In light of the capacity constraints on the Vietnamese side, the authorities think it appropriate to take a cautious and gradual approach – trial through piloting, followed by a nation-wide rollout.

330. They noted that the introduction of an LTO should be in incremental steps – the progress of setting up an IT system under the World Bank’s assistance also affects the sequence and enforcement of the LTO establishment. Such remarks by the authorities symbolize their strong ownership of the reform process – they are in the driver’s seat and have a clear vision, strategy and concrete approach to carry out reforms.

(4) Coordination with Other Development Partners 331. The Fund has provided assistance in developing new policy and framework in the reform process, whereas other development partners have mainly been providing “downstream” assistance on the concrete technical aspects to implement such reform through individual programs and projects. JICA, for example, has been providing assistance in taxpayer services, audit techniques, and developing the GDT’s capacity to provide training to its staff. The EC project which was recently completed, aimed to enhance capacity for raising domestic tax revenue for all major revenue sources. The World Bank has been contributing to the reorganization of the GDT, tax policy analysis and

32 According to the World Bank, the Multi-Donor Trust Funds (MDTF) managed by the Bank also provides funding to the Fund’s TA.


90

the Tax Administration Law, with which they view the Fund’s TA as complementary to their undertaking.

332. As mentioned above, the World Bank has also been assisting enhancement in the IT area to support core program areas and self assessment through the MDTF. The Vietnamese authorities, taking into consideration advice provided by the Fund’s experts, has been deciding what assistance to request from which organizations, based on their five-year strategic plan. Hence, the GDT is the focal point to ensure coordination of the delivery of assistance among development partners, and to make sure that the assistance is aligned with the GDT’s strategic plan.


333. The authorities were aware that the funding source of the Fund’s TA was from Japan, but they also mentioned that GDT staffs not involved in the Fund’s TA operations may not have such knowledge.

4.2.4 Tanzania ----------------------------------------------------------------------------------------------------------------------- Case of Good Practice – Tanzania (1) Fiscal Year: 2005 Project No. 3049 IMF TA Department: LEG Project Title: Central Bank and Banking Legislation Cost: $51,700 Recipient: Tanzania Local Counterpart: Governor of Central Bank of Tanzania -----------------------------------------------------------------------------------------------------------------------


334. In January 2004, the Governor of the Bank of Tanzania announced that Tanzania’s Authorities had decided to undertake a comprehensive review of the Bank of Tanzania Act (1995) and the Banking and Financial Institution Act (1991) with a view to updating and modernizing them. The authorities planned to introduce amendments to ensure that the financial sector’s legal framework conforms to best practices and international standards. In view of these objectives Tanzanian authorities requested technical assistance from IMF in the review and amendment of these two laws. A Project Proposal to apply JSA funding for this technical assistance was approved for two short term experts.


335. IMF sent a technical assistance (TA) team consisting of two experts financed by JSA and one IMF staff, for a period of one month in July 2004, to assist the authorities in revising their central bank and banking laws. The two JSA TA experts were Mr. Robert Sparve, former General Counsel of the Central Bank of Sweden who worked on the central bank part of the team, and Ms. Virginia Rutledge, an American lawyer with very extensive experience in the legal aspects of banking supervision and banking regulation, who worked


91

on the commercial banking part of the TA team.

336. Among the discussion with the authorities, the most intensive discussions were made on the topic of proposed amendments of the central bank act, especially regarding the independence of the Board and the governance of the institution. The TA team left draft amendments to the Bank of Tanzania Act (1995), and to the Banking and Financial Institution Act (1991) and these were sent to the authorities, after the TA team’s visit.


92


337. The input by JSA TA team experts made a significant contribution for bringing the financial sector’s legal framework in conformity with best practices and international standards. In 2005, Tanzanian authorities took over most part of the proposed amendments by IMF/JSA TA and submitted the revised laws to Parliament, and the Parliament adopted the amendments the same year.

338. During an interview with the evaluation team in August 2009, Mr. Juma Reli, Deputy Governor of the Bank of Tanzania mentioned that this JSA TA was very useful and effective in contributing to the modernizing of the legal framework of the central bank and banking sector of Tanzania. Mr. Reli also noted that bringing a Swedish ex-central banker and a US lawyer as TA experts could only be done by an IMF TA facility, in order to seek advice at the level of international best practice.

----------------------------------------------------------------------------------------------------------------------- Case of Good Practice – Tanzania (2) Fiscal Year: 2006 Project No. 3655 IMF TA Department: FAD Project Title: Public Financial Advisor Cost: $142,500 Recipient: East AFRITAC Local Counterpart: Ministries of Finance and Accountants General in East Africa -----------------------------------------------------------------------------------------------------------------------


339. The East Africa Regional Technical Assistance Center (East AFRITAC) of IMF was established in October 2002, to provide technical assistance in the areas of public financial management (PFM), macroeconomic policy, revenue policy/administration, economic and financial statistics, financial sector regulation and supervision. The regional coverage is for Eritrea, Ethiopia, Kenya, Rwanda, Tanzania and Uganda.

340. In recent years, East AFRITAC has been increasingly involved in technical and strategic support to PFM reform programs, partly as a consequence of donors’ assistance, in the region, shifting more and more towards general budget support.

341. This project provided financing to one of two PFM advisors based at the East AFRITAC, to provide technical support on the following PFM issues for the governments in the East AFRITAC region.

Improving the budget preparation and execution process Improving expenditure control monitoring Reforming treasury systems, upgrading cash and debt management procedures, and

modernizing financial accounting and reporting systems Formulating modern state financial management legislation Assisting targeted lower levels of government in the context of country fiscal

decentralization processes


93

Improving the internal structure and organization of finance ministries.


342. The JSA TA Expert, Mr. Vijay Ramachandran, undertook the following TA activities as a resident PFM advisor stationed at East AFRITAC.

Eritrea: assistance in drafting accounting manuals, review of the use of the GFS 2001 based classifications in the 2006 budget, and examination of the functioning of the cash management reforms.

Ethiopia: to review progress in the program budgeting TA being provided by East AFRITAC through a peripatetic advisor, and also to undertake a short review of the cash management reforms.

Kenya: to look at oversight of, and reporting from, semi-autonomous government agencies and extra-budgetary funds, as well as the use of Appropriations in Aid in the Government of Kenya.

Kenya: to enable AFRITAC participants in the Development Partners Group’s assessment of Kenya’s “Strategy for the Reform of PFM”

Rwanda: advice on the finalization of the financial regulations. This was followed up by the preparation of detailed instructions on bank reconciliations for the authorities to consider.

343. The expansion of East AFRITAC’s TA in the PFM area by two resident experts was a reflection of the need in TA in the area supported by the past achievement of the EAST AFRITAC’s TA in the previous years. Mr. Ramachandran, JSA Expert, worked together with the other long standing experts to accumulate knowledge and experience at East AFRITAC.


344. The Evaluation Team interviewed with Mr. Mario Zamaroczy, Coordinator and Mr. Stephen Mayes, current PFM Advisor at East AFRITAC, in August 2009. East AFRITAC is supported by 16 donors, and the donors provide direct funding to the East AFRITAC, except for Japan which is through the financing cost of the resident expert. Due to the nature of these donors support and contribution, it is not appropriate to single out the performance of the JSA financed expert as an evaluation target of JSA contribution, but rather to evaluate the East AFRITAC’s performance as one of 16 donors.

345. An independent evaluation of the AFRITAC's was recently conducted, and the report was released in March 200933. According to this report, AFRITAC’s are generally evaluated as good, while East AFRITAC is positioned in a leading role and the most successful among the AFRITAC’s. In particular the performance of Revenue Administration TA by East AFRITAC is rated at the high end of the good range.

33 “Final Report of the Independent Evaluation of the African Technical Assistance Centers”, by Bruce Murray (Team Leader) and three other members, published by OTM, IMF


94

346. Mr. John Michael Haul, Deputy Secretary General of Tanzania Ministry of Finance, who is one of the recipients of TA by East AFRITAC in the area of PFM, commented the TA was useful and effective in the area of revenue administration, legal aspects of PFM, budget system improvement, macroeconomic analysis, and fiscal supervision and statistics. For Tanzania, due to the closeness of its physical presence to East AFRITAC, the value of a resident advisor is particularly high, since various consultation issues are arising through daily work.

5. Overall Evaluation and Rating System

95

Chapter 5 Overall Evaluation Results and Proposed Rating System

5.1 Overall Evaluation Results

347. In this section, a consolidation of the evaluation survey results, from the Questionnaire Survey, Beneficiary Survey, Assessment Reports and interviews in the field visits, is attempted to provide an overall picture of the IMF/JSA TA evaluation.

5.1.1 Evaluation Results Based on OECD Evaluation Criteria

1) Relevance

348. The Questionnaire Survey conducted by the study team, for both the recipient countries and experts, clearly indicated that they recognize IMF/JSA TA projects as being relevant to recipient government needs and priorities and necessary timings. The answers for consistency with the recipient government policy were either “excellent” (64%) or “good” (36%), and there was no answer for ”modest” or “poor”, as shown below.

Answers by Beneficiaries for Consistency with Government PolicyAnswer Response Share


Modest 0 0 Poor 0 0

No knowledge 0 0

349. The field survey results indicate that IMF/JSA TA projects are relevant to the recipient countries’ developing priorities, based on their country contexts, and also relevant to IMF priorities, whose regional strategy is indicated in RSN's.

350. According to the interviews with TA departments at IMF headquarters, IMF TA projects are highly specialized and quite focused on the relevant development issues in monetary and finance policies, legislation, process and procedures which makes the TA highly relevant to other TA by other donors.

2) Effectiveness (Impacts)

351. Overall results of the evaluation surveys indicate that IMF/JSA TA projects are basically effective and achieved the desired goals. The results of the “Project Assessment” by TA Departments indicate that TA projects have been regarded as effective in light of the targeted objectives. The results of the “JSA Evaluation Questionnaire” (by the beneficiaries) indicated a 80% “yes” response regarding the usefulness of the TA, as shown below.


96

Answers by beneficiaries for usefulnessAnswer Response Share

Yes 25 80.6% Partially Yes 6 19.4%

352. Based on our questionnaire surveys for beneficiary countries and experts, IMF/JSA TA projects have mostly achieved their expected results, with most answers as “excellent” (39%) or “good” (62%), and with no answers for Modest or Poor. TA outputs have been used by authorities and contributed to improvement of knowledge, awareness and motivation for reforms among them. In addition, TA outputs, to some extent, have directly led to improvement of policies and their instruments in recipient countries. Major reasons for their effectiveness are: 1) expertise of advisors, 2) experience of experts in other countries, 3) communication skills of experts in understanding country context, and 4) backstopping by headquarters.

353. The field survey results show that IMF/JSA TAs are quite effective in generating (or contributing towards generation) of tangible results. Main reasons for the effectiveness are: 1) expertise of experts, 2) focus on competitive sectors, 3) selection method (roster system) of experts, 4) communication skills of experts, and 5) backstopping by IMF headquarters.

3) Efficiency

354. Based on our field survey results, IMF/JSA TAs seem to be efficient due to their well-focused approach and selection of competent experts. Overall evaluation results of the study are basically positive in regard to the efficiency of the IMF/JSA TA projects.

355. Regarding the efficiency in terms of cost, JSA is well managed through cost-effective means such as the roster-based selection of experts, smooth implementation supported by strong backstopping from headquarters, and repeated assignment by the same expert whose ability is proven. The budget per project and standard cost of long-term experts indicates that these are well within the reasonable range, compared with TA provided by Japanese ODA agencies and other international organizations.

356. The questionnaire survey results for JSA Experts indicate that in comparison with other international organizations, IMF/JSA TA is higher in quality by 92%, while the cost is higher by 46%, indicating comparative higher efficiency in terms of quality and cost consideration, as shown below.

Answers by Experts for comparison with other international organizationPoints of Difference Answered as “yes”

Quality is higher for IMF/JSA 92% Monthly remuneration is higher for IMF/JSA 46%


97

4) Sustainability

357. According to our questionnaire surveys for both recipient countries and JSA experts, major factors for sustainability of TA projects such as institutional memory, staff capabilities, or budget allocation are achieved in most of the counterpart authorities, as shown below. The majority of the experts have expressed their shared views that sustainability of IMF TAs depend on a willingness to reform by authorities.

Answers by Experts for levels of sustainabilityAnswer Response Share


Modest 11 13.4% Poor 1 1.2%

No knowledge 7 8.5%

358. The field survey results indicate that the sense of ownership by recipients varies by country, e.g. Vietnam with strong ownership. The concept of increasing the sense of ownership for countries with weak ownership needs to be addressed for the sustainability of the TA contribution.

359. As for financial sustainability, it is forecasted that IMF has to rely more on externally financed sources, because of decreasing internally funded sources. IMF has introduced a charging system for internally-funded TA, which might not be significant in terms of volume, but it will have an impact on enhancing ownership of the recipient countries. Therefore, an application of the charging system to JSA will be needed to be considered.

5) Transparency

360. Although the IMF/JSA TA projects deal with policy-level and highly specialized areas of finance and monetary systems, the field survey results indicate that there is room for improvement in sharing information with other donors, including Japan, regarding the activities of IMF/JSA TA projects.

361. The recruitment system for selecting experts for IMF TA is by a roster system, due to the limited number of projects and competent experts to be employed. Although the current system is considered to be reasonable for operational efficiency, enhancement in transparency of the recruitment system is needed.


362. Regarding the consistency of JSA with the Japanese ODA policy, three main documents compiled by the Japanese Government (ODA Charter, Mid-term Policy and Country Assistance Programs) were examined. The result of this examination was an assessment that IMF/JSA TA is highly consistent with the Japanese ODA policy.


98

7) Visibility of Japan as a Donor

363. The overall evaluation results indicate that the visibility of Japan as a donor for IMF/JSA TAs and is relatively low compared with the high scores gained for other aspects of evaluation and for the effectiveness and benefit for the recipients.

364. The Questionnaire Survey results show that Japanese sponsorship JSA TAs is known by 71% of the high ranking officials in the recipient governments. On the other hand, 96% of experts answered that they are aware of the Japanese sponsorship. The narrative comments by experts indicated their support and understanding on the need for enhancing visibility of Japan as a donor.

365. The results of the field survey revealed that in particular countries, e.g. Cambodia, visibility is very high on the recipient side as well, partly because of the volume of JSA TA delivered and the IMF-Japan (MOF) joint mission’s visit in early 2009. However, in other countries visited, e.g. Indonesia, Tanzania and Vietnam, the awareness on the recipient authorities was low, suggesting some room for improvement.

Answers by beneficiaries for recognition of Japan as a donorAnswer Response Share

Yes, very much so 12 30.8% Yes, relatively so 13 33.3%

Not so much 5 12.8% Not at all 1 2.6%


5.1.2 Conclusion: Evaluation Results Based on TOR criteria

1) Overarching Objectives

366. The overarching objectives are defined by the three points below. (1) To assess whether the JSA has been an effective instrument for capacity building in

countries, covering issues of (a) quality of TA, (b) sustainability, and (c) efficiency; (2) To determine whether the administration of and information on the JSA is sufficient to

provide accountability and whether the JSA provides good value for money, covering issues of (a) consistency with Japanese ODA, (b) visibility of Japan as a donor, (c) supporting reasons for IMF TA; and

(3) To make recommendations to improve the JSA in furtherance of the objectives in (1) and(2) above.

2) The JSA as an Effective Instrument for Capacity Building in Countries

367. The followings are findings by the evaluation team for each of the subjects being questioned.


99

(a) Quality of JSA TA 368. Is the JSA TA of high technical quality consistent with international best practice and member country priorities?

Most of the recipient country counterparts recognize that the quality of IMF/JSA TAs is at the best international level.

Most of the recipient countries also agree that IMF/JSA TA's are quite consistent with their developing priorities.

369. Is the involvement of area departments useful in determining member country TA priorities, integrating them with Fund surveillance and lending activities, and providing quality control?

The involvement of area departments is useful and critical because initial identification of needs is done by the area departments. They monitor surveillance results and consider what kind of TA activities and areas are necessary. Area departments compile Regional Strategy Notes (RSN) which outlines regional issues to be addressed and assistance priorities and plans in each country.

370. What measures are in place in TA departments to ensure that IMF/JSA TA is timely and of high quality? Have these measures been adequate/effective?

TA departments provide a strong “back stopper” service, which is key for quality control of JSA and other TA activities. The headquarters staff in TA departments are specialists themselves, who have deep knowledge and experience in the subject areas. This backstopping function differentiates IMF TA from other donors, where usually, the generalist staff manages TA activities. As for timeliness of TA, JSA is a quick funding source compared with other externally funded TAs.

371. Are the current evaluations/assessments of JSA activities adequate/effective? The current evaluation/assessment system of the IMF/JSA TA seems to have much

room for improvement from the viewpoints of both “accountability” and “management improvement.”

(b) Sustainability of Results Achieved Through IMF/JSA TA 372. Have JSA activities as stand-alone, or in conjunction with IMF financed activities, achieved tangible and sustainable results in beneficiary countries?

Most recipient country counterparts recognize that the implemented IMF/JSA TAs have been able to generate tangible outcomes, in some cases impacts, in their countries.

The “coverage” of IMF/JSA TA, however, varies with countries quite widely.

(c) Efficiency and Cost Considerations 373. Is IMF/JSA TA a cost-effective way to deliver capacity building assistance to countries, especially in relation to other comparable TA delivery modes (as determined by the evaluators), and bearing in mind the difficulties inherent in measuring the benefit of capacity building activities?

Since many (most) recipient countries regard the contents of IMF/JSA TA as unique, the comparison with other developing partners seem not to be relevant.

However, if we dare make some comparison, there seem to be several factors which


100

contribute to the cost-efficiency of JSA. First, the roster-based selection process of experts is more cost-efficient than public bid or open application procedures. Second, the repeated assignment of the same qualified expert whose ability is proven, is efficient in many respects due to the established relationship and experience in the field. Third, strong support and quality control of IMF headquarters’ functional departments contribute to the smooth and efficient implementation of IMF/JSA TA. The standard cost of dispatching the expert, as far as the remuneration rate is concerned, seems to be reasonable compared with similar Japanese TA schemes.

374. Is the cost of IMF/JSA TA reasonable in relation to other comparable TA providers? The Budget per project (around US$150,000) and the standard monthly remuneration

rate for dispatching experts as determined in the JSA Guidelines seems to be reasonable compared with similar cases in Japanese bilateral technical assistance such as JICA. Furthermore, the disbursement rate is quite high, 96% on average from 1993 to 2009, which is another aspect of sound management and administration by the IMF department in charge.

3) IMF/JSA TA Delivery is Providing Good Value for Money for Japanese Taxpayers

(a) Consistency of JSA Activities with Japan’s ODA policy 375. Have JSA activities been consistent with Japan’s ODA policy?

There are three major documents compiled by the Japanese Government which represent the Japanese ODA policy, i.e. ODA Charter, Mid-term Policy and Country Assistance Programs. Examination of these documents show that IMF/JSA TA activities are highly consistent with the Japanese ODA policy.

376. Have the JSA activities overlapped negatively with other Japanese ODA initiatives, e.g., bilateral ODA and/or activities of “Japan funds” at other international institutions? If so, how can these overlaps be best removed/addressed?

None of our findings indicate the existence of duplication or overlap between the IMF/JSA TA and other Japanese ODA technical assistance implemented. This is partly due to the difference in areas of specialty focused on by the IMF and Japanese ODA. Nevertheless, there is an important and necessary complementary role for IMF/JSA TA in light of future development of Japanese bilateral ODA in the financial and fiscal sectors.

(b) Visibility of Japan as a Donor 377. Has the JSA provided adequate visibility to Japan as an ODA donor country?

Partly yes and partly no. There is room for improvement, in particular on the beneficiary side, with more institutionalized arrangements by the IMF and the Japanese authority in charge.

378. Has the JSA enhanced the visibility of Japan as an ODA donor country? Partly yes and partly no. Since Japan is recognized as a large donor, in particular in

Asia, the effectiveness of enhancing visibility through JSA is, if properly implemented, is marginal. However, in countries where the Japanese presence is not strong, e.g.


101

Africa, utilization of JSA, as a part of IMF which has strong influence in the financial and fiscal sector, might have more effectiveness.

379. Have the “publicity” guidelines in the JSA been effective? If not, how can they be changed to improve the visibility of Japan as an ODA donor country?

There seem to be various possibilities for improving Japan’s visibility including publicity activities by building into administrative process. This is discussed under recommendations in the next chapter.

(c) Support for IMF TA Versus other Avenues for Providing Development Assistance 380. What are the advantages of supporting IMF TA, as compared to using Japanese funds for other development assistance?

For Japan, the areas involving financial systems, fiscal management and macroeconomic analysis are areas with limited human resources that need to be supplied in TA projects to developing countries. The advantage of sponsoring IMF TA through JSA is that it expands the opportunity for Japanese bilateral ODA coverage, with possible collaboration and complementary exchange among experts and ODA agencies; typically JICA. IMF TA in African countries. e.g. Tanzania, is regarded in Japan as a pioneering effort, compared to some countries in Asia, e.g. Vietnam, where the country is already utilizing JICA TA in the central banking and fiscal areas. A strategy can be implemented to make use of JSA for pioneering efforts in a less experienced country. The coverage of bilateral donors will increase when Japanese ODA catches up in providing its own TA directly to the recipient countries. (see the illustration of this concept below)

Chart 5.1 Complementary Role of IMF TA and Bi-lateral TA


102

5.2 Proposed Rating System

5.2.1 Proposed “Rating” System in Brief

381. The evaluation team proposes a “rating” system for IMF/JSA TA projects for future implementation. The external evaluators could apply the rating system to their ex-postevaluations as a major potential use of the system. The proposed system is developed with reference to several precedents of IMF TA evaluation studies (Iraq in 2008, and AFRITAC in 2009) as well as guidelines for TA evaluation prepared by the ADB in 2006. Among these, the evaluation study on Iraq is quite similar to this evaluation study, and therefore, basic ideas are taken from the Iraq report.

1) Rating Criteria, Sub-Criteria, and Weight

382. Based on the OECD “DAC Criteria for Evaluating Development Assistance,” the rating criteria are basically composed of “relevance,” “effectiveness,” “efficiency,” and “sustainability.” Viewpoints on “impact” are included within the “effectiveness” criterion. Precedents and guidelines referred to in this report also use the same criteria. In addition to these basic criteria, specific criteria related to donor concerns are also included.

383. In each rating criterion, sub-criteria and their weightings are studied by making comparison to referenced systems and guidelines. In terms of “relevance,” sub-criteria and weight are the same as the IMF-Iraq (2008) evaluation. For “effectiveness,” sub-criteria are primarily from two evaluation reports done for IMF TA projects. We propose to add one more sub-criteria, “beneficiary satisfaction,” which is also an important factor for the TA projects’ effectiveness, and the necessary data are available for the IMF/JSA TA projects. Weighting for the 4 sub-criteria is even, taking into account their equal importance for achieving desired impacts on the economic situation of recipient countries. With regard to the “efficiency,” sub-criteria from the Iraq study are basically utilized, though we excluded “process and implementation efficiency” due to the relative difficulty of objective ratings. Weighting for the 3 sub-criteria is close to even, since all of these are equally necessary for maintaining high efficiency. For “sustainability,” we propose to use 2 sub-criteria from the ADB guideline that specifically divides the concept of “ownership” into several sub-factors. Those are also relatively easy to make judgments from. In addition to these sub-criteria, a criterion related to the external factors is included. Since ownership is really key for sustainability, relatively high weights are put on the first 2 sub-criteria.

384. The evaluation team proposes to add two criteria related to donor specific issues. Specifically, they are “donor relevance,” and “donor effectiveness.” The sub-criterion under “donor relevance” is “consistency with donor priorities,” and under “donor effectiveness” is “visibility of donor.”

385. These sub-criteria and their weighting could be modified in order to appropriately reflect the exact situation of the TA projects, through their trial applications.


103

Chart 5.2 Criteria, Sub-criteria and Weights<Relevance>

Sub-criteria Weight Consistency with government priorities & specific needs 30% Consistency with IMF priorities (“Regional Strategy Notes”) 30% Quality of “diagnostics” 20% Coordination with other developing partners 20%

<Effectiveness>Sub-criteria Weight

Desired vs. Actual outputs 25% Use of outputs 25% Significance of contribution to developing core economic functions/institutional building

25%

Beneficiary satisfaction 25%

<Efficiency>Sub-criteria Weight

Timeliness 35% Efficient Use of resources (No excess to original budget) 35% Consistency of engagement 30%

<Sustainability>Sub-criteria Weight

Appropriate policies, procedures, and financial structures in place to ensure continued operation of partner organizations

40%

Appropriate policies in place to ensure the maintenance of required human resources in partner organizations

40%

Country context (external factors) 20%

<Donor Relevance>Sub-criteria Weight

Consistency with donor priorities -

<Donor Effectiveness>Sub-criteria Weight

Visibility of donor -


104

Chart 5.3 Comparison with Referenced Systems & Guidelines<Relevance>

ADB (2006) IMF-Iraq (2008) IMF-AFRITAC (2009) JSA (2009) - Evidence of diagnostic

assessment of problems and opportunities

- Consistency with government’s priorities and strategy

- Consistency with ADB’s institutional objectives, and strategy and program for the country

- TA formulation/design - Evidence of

participation or government ownership

- Degree of coordination and/or complementarily with activities of development partners

- Quality of diagnostics (20%)

- Consistency of TA with IMF activities in Iraq (30%)

- Consistency with government priorities (30%)

- IMF/development partner coordination (20%)

- Consistency with government priorities (60%)

- Coordination with developing partners (20%)

- Consistency with IMF headquarters activities (20%)

- Consistency with government priorities & specific needs (30%)

- Consistency with IMF priorities (Regional Strategy Notes) (30%)

- Quality of “diagnostics” (20%)

- Coordination with other developing partners (20%)

<Effectiveness> ADB (2006) IMF-Iraq (2008) IMF-AFRITAC (2009) JSA (2009)

- Effectiveness of TA management

- Achievement of TA outcome, including institutional development (where appropriate)

- Attribution

- Use of outputs (30%) - Desired vs. Actual

outputs (30%) - Effectiveness of offsite

delivery mode (10%) - Significance of IMF’s

contribution to developing core economic functions/institutional building (30%)

- Use of AFRITAC Outputs (40%)

- Planned vs. Actual achievements (30%)

- Significance of contribution to developing core economic functions and institution building (30%)

- Desired vs. Actual outputs (25%)

- Use of outputs (25%) - Significance of

contribution to developing core economic functions/institutional building (25%)

- Beneficiary satisfaction (25%)

<Efficiency> ADB (2006) IMF-Iraq (2008) IMF-AFRITAC (2009) JSA (2009)

- Achievement of TA outcome with efficient use of inputs

- Efficiency of process - Timeliness of

outcomes - Quality of mid-course

adjustments and dialogue

- Process and implementation efficiency (30%)

- Consistency of engagement (30%)

- Efficient use of Resources (30%)

- Timeliness (10%)

- Process and implementation efficiency (40%)

- Efficient use of resources (40%)

- Monitoring and reporting (20%)

- Timeliness (35%) - Efficient use of

resources (No excess to original budget) (35%)

- Consistency of engagement (30%)

<Sustainability> ADB (2006) IMF-Iraq (2008) IMF-AFRITAC (2009) JSA (2009)

- Appropriate policies, procedures, and financial structures in place to ensure continued operation of partner organizations

- Appropriate policies in place to ensure the maintenance of required human resources in partner

- Iraqi context (20%) - Continued IMF

involvement (30%) - Government

ownership of TA outputs (30%)

- Likelihood that participants will apply the knowledge gained on the job (20%)

- Political economy issues (25%)

- Executing agency ownership and use of the outputs (40%)

- Promoting the use of African expertise (15%)

- Financial sustainability of the centers (20%)

- Appropriate policies, procedures, and financial structures in place to ensure continued operation of partner organizations (40%)

- Appropriate policies in place to ensure the maintenance of required human


105

ADB (2006) IMF-Iraq (2008) IMF-AFRITAC (2009) JSA (2009) organizations

- Adequacy or otherwise of policies, institutions, markets and regulatory conditions, and risks of change

- Political will to ensure government or enterprise ownership and commitment to TA outcomes

- Existence of appropriate legislative structure to support the implementation of TA outcomes

- Political will on the part of government to maintain support of key stakeholders

resources in partner organizations (40%)

- Country context (external factors) (20%)

2) Scoring

386. Each sub-criterion is rated on a “4 point” scale respectively, then they are aggregated using the weights assigned to each sub-criterion. The scoring method is exactly the same as evaluations of TA in Iraq and AFRITAC.

Chart 5.4 Rating Scores<Relevance> - Highly relevant, or excellent: >3.5 - Relevant, or good: 2.5-3.5 - Partly relevant, or modest: 1.5-2.5 - Not relevant, or poor: <1.5

<Effectiveness> - Highly effective, or excellent: >3.5 - Effective, or good: 2.5-3.5 - Partly effective, or modest: 1.5-2.5 - Not effective, or poor: <1.5

<Efficiency> - Highly efficient, or excellent: >3.5 - Efficient, or good: 2.5-3.5 - Partly efficient, or modest: 1.5-2.5 - Not efficient, or poor: <1.5

<Sustainability> - Highly sustainable, or excellent: >3.5 - Sustainable, or good: 2.5-3.5 - Partly sustainable, or modest: 1.5-2.5


106

- Not sustainable, or poor: <1.5

<Donor Relevance> - Highly consistent, or excellent: >3.5 - Consistent, or good: 2.5-3.5 - Partly consistent, or modest: 1.5-2.5 - Not consistent, or poor: <1.5

<Donor Effectiveness> - Highly visible, or excellent: >3.5 - Visible, or good: 2.5-3.5 - Partly visible, or modest: 1.5-2.5 - Not visible, or poor: <1.5


107

5.2.2 Information/Data Sources and Necessary Improvement for Appropriate Rating

387. The table below shows the hypothetical information/data sources for the “rating” practice. In order for the appropriate rating, some improvement among the sources is necessary. As discussed in Chapter 3.1, possible improvements in the evaluation reporting system of the JSA-funded TA project would enable stable evaluation and rating practices in the future.

Chart 5.5 Information/Data Sources for “Rating” System

Rating Sub-criteria Information/Data Sources Index

for Improvement

<Relevance> Consistency with government priorities & specific needs

Project Proposal, Project Assessment

Consistency with IMF priorities (“Regional Strategy Notes”) Regional Strategy Notes

Quality of “diagnostics” Project Assessment (B) Coordination with other developing partners

Project Proposal, Project Assessment

<Effectiveness>Desired vs. Actual outputs Project Assessment (C)

Use of outputs Project Assessment, (next) Project Proposal

Beneficiary satisfaction JSA Evaluation Questionnaire, Project Assessment

(C) (B)

Significance of contribution to developing core economic functions/institutional building

Project Assessment, (next) Project Proposal

(C) (B)

<Efficiency> Timeliness Project Assessment (C) Efficient Use of resources (No excess to original budget) Project Assessment (B, C)

Consistency of engagement Project Assessment (B) <Sustainability> Appropriate policies, procedures, and financial structures in place to ensure continued operation of partner organizations


(B) (B)

Appropriate policies in place to ensure the maintenance of required human resources in partner organizations


(B) (B)

Country context (external factors) Project Assessment, (next) Project Proposal

(B,C) (B)

<Donor Relevance>

Consistency with donor priorities Policy and strategy document(s) prepared by Japanese Government

(A)

<Donor Effectiveness>

Visibility of donor JSA Evaluation Questionnaire, Project Assessment

(B) (B)


108

388. Generally, there seem to be three types of possible improvements in regard to the information/data sources, as described below. The characters (A) (B) (C) attached to source names in the table represent those three types respectively:

(A) Appropriate documents themselves are necessary when stakeholders think policies are important;

(B) Specific documents themselves already exist. However, these sub-criteria should be tangibly incorporated within documents in order to secure necessary information; and

(C) Specific documents with necessary sub-criteria already exist. However, implementing practices should be consistent as well as universal, so that necessary information for the rating is always available with stable quality and quantity.

6. Recommendations to IMF and MOF

109

Chapter 6 Recommendations to IMF and MOF

6.1 Recommendation to IMF

6.1.1 Improvement in Transparency

389. It is recommended that the results of project-level evaluation by recipients, collected through JSA evaluation questionnaires, should be summarized and reported with more detailed analysis of the evaluation results in the JSA Annual Report in order to present this information to the general public both in Japan and in other countries.

390. In order to maintain the current recruitment system which is considered to be operating efficiently, it is recommended that there be disclosure of the recruitment process for experts to be employed by the IMF for JSA TA. Particular emphasis should be placed on the specific qualifications required, relevant background and experience along with communication skills with recipient agencies and IMF headquarters. This will help to increase transparency in the recruitment procedure since the recruitment process does not utilize an open-bid system similar to other donor agencies.

391. It is recommended to institutionalize information sharing with other donors regarding the contents of TA activities. This will provide enhanced transparency and possible complementary effect with the other donors’ input, in particular with Japan, i.e. the Japan International Cooperation Agency (JICA). Since there might be a limitation in information disclosure with the authorities in the recipient countries, the level and content of information sharing must be considered in light of country and sector context as well as potential interest by other donors in the field. The IMF Resident Representative should assume a coordinating function in the field.

6.1.2. Improvement of Management

392. It is recommended that management of JSA Project Proposals and Project Assessments be improved regarding the points listed below:

Improvement in consistency in the information to be filled-in; Improvement in filling out “verifiable indicators” for the projects with guidance regarding

the intention and the concept; Improvement in describing the assumptions for the project; Improvement in filling in the matrix for the mid-term objectives; Improvement in describing the achievement of the projects and their intended targets; Improvement in procedures regarding the extension of projects with conditions to be

clarified.


110

393. In order to enhance the effectiveness, impact, and efficiency of project, it is recommended that a strategy should be established for programming the TA project in combination with long-term and short-term (peripatetic) experts, depending on the level of focus for a particular issue in a particular country, with attention toward:

More focus regarding both on the “IMF Context” and the “Country Context” in selecting the country and areas of TA.

An exit policy and targets that need to be achieved within a limited period. This will enhance ownership of the recipient, which is also indicated in the 2005 IEO Report as a need for graduation of long-term experts.

394. It is recommended to focus more on “sustainability” and “ownership” of the recipient organizations by incorporating them in the JSA proposals as well as in the Project Assessment, through applying methodologies in objective evaluations.

395. It is recommended that OTM improve the managing and administering JSA Project Proposals, Project Assessments, and JSA evaluation questionnaires, through a more efficient system to store, file, collect, and sort data on the IMF intranet. While we discovered that OTM started its improved system for collecting the questionnaires, more attention is necessary to sustain the effort given the limited resources in OTM.

6.1.3. Improvement of Visibility


396. It is recommended that a logo and/or concept be printed in support of the JSA and this logo should appear on IMF/JSA TA relevant materials, documents, reports and name cards of all the experts, to indicate clearly that JSA is being financed by Japan.

2) Enhance Visibility of the Recipients

397. As the results of this evaluation study indicate, the recipient authorities need to be more informed of the Japanese contribution to IMF/JSA TA projects. In this regard, it is recommended that the IMF enhance its procedures so that the recipients of JSA TA, at various levels, are well-informed of Japan as the donor.

3) Enhance Japan’s involvement in JSA Projects

398. It is recommended that the recruitment of Japanese experts for short-term assignment for seminars and workshops be strengthened, especially for assignments of less than one week. While the current number of Japanese experts in the roster is 30 out


111

of a possible 1,350 in total as of June 2009, cooperation with Japanese authorities is needed to up-date and improves the roster.

399. It is recommended that regional forums are organized and conducted in order to introduce activities covered by IMF/JSA TA in Asia and/or Africa. These forums are to be held in Japan, with the participation of IMF experts, recipient governments, The Japanese Government, ODA agencies, Japanese experts, scholars, the private sector and the mass-media. This will facilitate the: (i) dissemination of information focused on relevant experience, (ii) publicizing JSA’s financing support, and (iii) strengthening communication with the Japanese ODA circle.

6.2 Recommendations to MOF

6.2.1. Improvement of Accountability and Information Disclosure

400. It is recommended that public relations should be strengthened in order to relay information on JSA projects and their achievements to the general public, i.e. taxpayers, through seminars, forums, websites, the writing of articles on ODA publications and other media.

401. It is recommended that Information sharing should be enhanced regarding Japanese ODA agencies and relevant groups, academic circles, experts and consultants, with a focus on project-level information for JSA projects. This will lead to the identification of interesting issues to be covered by the proposed regional forum.

402. It is recommended that a strategy is created to make use of IMF TA and the interests of Japanese ODA in the field of financial and fiscal system development in developing countries with attention to:

Policy of MOF and MOFA for ODA policy, utilization of Japanese resources and utilizing IMF capacity in a complementary arrangement.

Strategy to use the JSA as a tool to supplement Japanese ODA. Develop documents to explain the Japanese historical experience in monetary and

fiscal systems, and policy implementation experience as reference material for developing countries.

Establish a strategy to mobilize JSA and JICA experts, depending on the country context in development needs of the target countries’ financial and fiscal system, and possible involvement of Japanese bilateral assistance.


112

6.2.2. Improvement of Visibility


403. It is recommended that cooperation with IMF occur in the symbolic improvement of JSA related documents to be used by IMF staff as well as experts. In order to have the recipient government to be aware of the Japanese assistance, the use of bilateral channels to inform the recipient government can be effective.

2) Enhance Japan’s Involvement in JSA projects

404. It is recommended that the number of Japanese experts on the IMF roster of experts be increased with screening by capacity and appropriate tasks to be covered. Despite the efforts by IMF and MOF, in the past to increase Japanese participation, use of Japanese nationals for IMF/JSA TA is quite limited. In order to improve the participation of Japanese experts, more participation of officials from MOF, Bank of Japan and Financial Service Agencies should be encouraged with cooperative effort by the respective departments among Japanese agencies in seeking capable staff as IMF/JSA TA experts.

405. In order to increase awareness by Japanese domestic interested parties, it is recommended regional forums be organized to disseminate information on the activities of JSA experts, in particular, with countries that have limited information about the Japanese ODA circle.

406. It is recommended that a system to have JSA experts establish communication with JICA at headquarters as well as field offices should be set up. This could include possible collaboration in the field.

Offi ce of Technical Assistance Management

700 19th Street NWWashington, DC 20431USATel.: 1-202-623-7646Fax: 1-202-623-7106Email: [email protected]