ENVIRONMENTALLY SOUND MANAGEMENT AND FINAL … · The Steel Authority of India (SAIL)’s Bhilai Steel Plant (BSP), is the lead beneficiary of this project. SAIL, Under the Ministry

UNITED NATIONS INDUSTRIAL DEVELOPMENTORGANIZATION

INDEPENDENT MID TERM EVALUATION

UNIDO PROJECT ON

ENVIRONMENTALLY SOUND MANAGEMENT AND FINAL DISPOSAL

OF PCB’S

INDIA

March 2015

Table of Contents

Table of Contents ........................................................................................................ 2 Acknowledgement ....................................................................................................... 5 Glossary of Evaluation Terms ..................................................................................... 7 List of Acronyms.......................................................................................................... 8 Executive summary ..................................................................................................... 9 1 Evaluation objectives, methodology and process ................................................ 12

1.1 Information on the evaluation ........................................................................ 12 2 Scope and objectives of the evaluation ............................................................... 13

2.1 Information sources and availability of information ........................................ 13 2.2 Methodological remarks, limitations encountered and validity of the findings 13

3 Country and project background .......................................................................... 14 3.1 Country context ............................................................................................. 14 3.2 Socioeconomic overview............................................................................... 14

3.2.1 Social overview ...................................................................................... 14 3.2.2 Economic profile ..................................................................................... 15

3.3 Policy and Legal framework: ......................................................................... 16 3.3.1 Investment climate ................................................................................. 16 3.3.2 Legal framework for environmental protection ........................................ 17

3.4 Environment Overview: ................................................................................. 18 3.4.1 Key National Environmental regulations in India - ................................... 18 3.4.2 Environmental Concerns in INDIA .......................................................... 19

3.5 Sector-specific issues of concern .................................................................. 19 3.5.1 Indian and Persistent Organic Pollutants (PoPs) .................................... 19 3.5.2 Institutions involved in POPS Management in INDIA .............................. 19 3.5.3 PCBs and India ...................................................................................... 20 3.5.4 Priorities and Action plan strategies ........................................................ 21

4 Project summary ................................................................................................. 22 4.1 Project Fact Sheet ........................................................................................ 22 4.2 Brief description ............................................................................................ 22

4.2.1 Project Objectives .................................................................................. 23 4.2.2 Project Outcomes and Outputs ............................................................... 23

4.3 Project implementation ................................................................................. 25 4.4 Positioning of the UNIDO project .................................................................. 26 4.5 Counterpart organization(s) .......................................................................... 27

2

5 Project assessment ............................................................................................. 28 5.1 Design .......................................................................................................... 28 5.2 Relevance ..................................................................................................... 29 5.3 Effectiveness ................................................................................................ 31 5.4 Efficiency ...................................................................................................... 39 5.5 Sustainability ................................................................................................. 41 5.6 Assessment of M&E systems ........................................................................ 41 5.7 Monitoring of Long Term Changes ................................................................ 42 5.8 Ratings Overview – Project Performance as per GEF Criteria ..................... 42 5.9 Assessment of processes affecting attainment of project results................... 43 5.10 Project Coordination and Management ....................................................... 45 5.11 Gender Mainstreaming................................................................................ 45 5.12 Procurement Issues .................................................................................... 45

6 Conclusions and Recommendations ................................................................... 46 7 Lessons Learnt .................................................................................................... 48 8 Annexes .............................................................................................................. 49

8.1 GEF Ratings ................................................................................................. 50 8.2 List of Interviewees ....................................................................................... 52 8.3 Bibliography .................................................................................................. 53 8.4 Terms of Reference for the MTE ................................................................... 54

9 I. Project Background and overview .................................................................... 56 10 II. Scope and Purpose of the Evaluation ............................................................ 59 11 III. Evaluation Approach and Methodology ........................................................ 60 IV. Evaluation Team Composition .......................................................................... 61 V. Time Schedule and Deliverables ........................................................................ 62 VI. Project Evaluation Parameters .......................................................................... 62 VII. Reporting ......................................................................................................... 68 VIII. Quality Assurance ........................................................................................... 69

11.1 Annex 1 - Outline of an In-Depth Project Evaluation Report ........................ 70 11.2 Annex 2 - Overall Ratings Table ................................................................. 72 11.3 Annex 3 - GEF Minimum Requirements for M&E ........................................ 75 11.4 Annex 4 – Required Project Identification and Financial Data ..................... 76 11.5 Annex 5. Job Descriptions .......................................................................... 80

11.5.1 Independent Evaluation of UNIDO project: “Environmentally Sound Management and Final Disposal of PCBs in India” ............................................. 83 11.5.2 GFIND10001 SAP ID: 104044 .............................................................. 83

11.6 Annex 6 – UNIDO Procurement Process .................................................... 86

3

4

Acknowledgement The evaluation team would like to acknowledge the many and diverse contributions made to this evaluation. We are particularly thankful to staff of UNIDO at Headquarters and in the field and, to officers and staff from government agencies and the private sector in the Republic of India interviewed during the field mission.

Cristóbal Vignal International consultant and Team leader

Hemant Verma National consultant

5

The designations employed and the presentation of material in this publication do not imply the expression of any opinion whatsoever on the part of the Secretariat of the United Nations Industrial Development Organization concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries.

Mention of company names and commercial products does not imply the endorsement of UNIDO.

The views and opinions of the Evaluation Team do not necessarily reflect the views of the Government of India, nor of UNIDO.

This document has not been formally edited

6

Glossary of Evaluation Terms

Term Definition

Baseline The situation prior to an intervention, against which progress can be assessed.

Effect Intended or unintended change due directly or indirectly to an intervention

Effectiveness The extent to which the objectives of a development intervention were or are expected to be achieved.

Efficiency A measure of how economically inputs (through activities) are converted into outputs

Impact Positive or negative, intended or non-intended, directly and indirectly, long term effects produced by a development intervention

Indicator Quantitative or qualitative factors that provide a means to measure the changes caused by an intervention

Intervention An external action to assist a national effort to achieve specific development goals

Lessons learned Generalizations based on evaluation experiences that abstract from specific to broader circumstances

Logframe (logical framework approach)

Management tool used to guide the planning, implementation and evaluation of an intervention. System based on (Management by Objectives) also called Results-based Management principles.

Outcomes The achieved or likely effects of an intervention»s outputs.

Outputs The products in terms of physical and human capacities that result from an intervention

Relevance The extent to which the objectives of a development intervention are consistent with beneficiaries requirements, country needs, global priorities and partners and donor»s policies

Risks Factors, normally outside the scope of an intervention, which may affect the achievement of an intervention»s objectives

Sustainability The continuation of benefits from an intervention, after the development assistance has been completed.

Target groups The specific individuals or organizations for whose benefit an intervention is undertaken

7

List of Acronyms

BAT Best Available Technology BEP Best Environmental Practices BSP Bhilai Steel Plant CPCB Central Pollution control Board CPRI Central Power Research Institute DDT Dichloro Diphenyl Trichloroethane DEA Department of Economic Affairs EAP Environmental Action Programme ESM environmentally sound management ET Evaluation Team FSP Full Scale Project GDP Gross Domestic Product GEF Global Environment Facility GoI Government of India Km Kilometer M&E Monitoring & Evaluation MoEF&CC Ministry of Environment, Forest and Climate Change MTE Mid Term Evaluation NIP National Implementation Plan PCBs Polychlorinated biphenyls PIF Project Identification Form PMS Project Monitoring System PMU Project Management Unit POPs Persistent Organic Pollutants PPG Project Preparation Grant PSC Project Steering Committee PVC Poly Vinyl Chloride RENPEP Regional Network on Pesticides for Asia and the Pacific SAIL Steel Authority of India Limited UNCTAD United nations Conference on Trade and Development UNIDO United Nations Industrial Development Organization WTO World Trade Organization

8

Executive summary This document presents the Independent Mid Term Evaluation (MTE) of the “Environmentally Sound Management and Final Disposal of PCBs in India” project, implemented in 2010-2014. The independent MTE was carried out during October and November of 2014 and was based on a review of all available literature and official project documents, semi structured interviews and discussions with key stakeholders and government authorities, and meetings with the project staff at headquarters and representatives in the field. The MTE mission took place from 27-31st October 2014.

The overall objective of this Global Environment Facility-funded (GEF) Full-Sized Project (FSP) is to reduce and eliminate the use and releases of PCBs to the environment through promotion of measures to minimize exposures and risks by introducing environmentally sound management and disposal of PCBs, PCB-containing equipment and PCB-containing mineral oils and wastes aiming at the final and virtual disposal of all PCBs inventory in India by 2025 and 2028, respectively. The immediate objectives of the project are to:

• Strengthen the legal and regulatory framework for environmentally sound management (ESM) and disposal of PCBs, PCB-containing equipment and PCB-containing mineral oils and wastes;

• Improve institutional capacity at all levels of PCBs disposal management; Removal of 7,700 tones of PCBs, PCB-containing equipment and PCB-containing mineral oils and wastes from targeted sites and transport them to disposal unit; and

• Disposal of 7,700 tones PCBs, PCB-containing equipment and PCB-containing mineral oils and wastes in an environmentally sound manner

The Ministry of Environment, Forests and Climate Chance (MoEF&CC) is the national executing partner for the project. The national executing agency of this project is the Central Power Research Institute (CPRI), under the Ministry of Power (MOP). CPRI is the government entity that developed and formulated the PCB-related chapters in the National Implementation Plan (NIP). The Steel Authority of India (SAIL)’s Bhilai Steel Plant (BSP), is the lead beneficiary of this project. SAIL, Under the Ministry of Steel, joined the project in its 1st year. SAIL is hosting the project implementation site at BSP, with a co-financing support estimated over Rs. 140 Crore (~ USD 23.3 mn) in the form of cost of land, civil structure, raw materials, utilities, required manpower and operating expenses for the facility.

UNIDO acts as the GEF Implementing Agency (IA) for the project and is responsible for the supervision of project management, facilitates stakeholder co-ordination, monitoring of the project, and reporting on project performance to the GEF. UNIDO is in charge of procuring the international expertise and equipments needed to deliver the outputs planned under the project components.

The Republic of India signed the Stockholm Convention on POPs on 14 May 2002 and ratified it on 13 January 2006. The Government of India (GoI) has identified this project amongst its immediate priorities for the implementation of the NIP, which was prepared and released in 2011, as it addresses national priorities.

Objectives, Scope and Methodology

The objectives of the MTE were to enable the GoI, GEF, UNIDO and other stakeholders and donors to: (a) verify prospects for development impact and sustainability (providing an analysis of the attainment of global environmental

9

objectives, project objectives, delivery and completion of project outputs/activities, and outcomes/impacts based on indicators); (b) enhance project relevance, effectiveness, efficiency and sustainability (by proposing a set of recommendations with a view to ongoing and future activities).

The key question for this MTE was to understand whether the project has made a significant contribution towards achieving its set objectives towards reducing the effects of POPs on human health and the environment, and rate the project progress on standardized evaluation parameters

Key Findings The design is considered to have been adequate to address the projects set objectives, however participation and involvement of the private sector and of civil society were found to be lacking. As well, the design did not properly factor in time constraints required to develop and establish a regulatory framework.

The projects is considered to be highly relevant as regards the overarching national priorities and it is also clearly aligned with those of the industrial sector, and the population as a whole. It is also relevant to GEFs operational programme and to UNIDOs thematic priorities.

A regards effectiveness, the ET assessed this against the stated outcomes and concluded this to be at this stage moderately unsatisfactory. However, as in pointed out in the report, this was due in great part to a series of unforeseeable events, which delayed results, as well as to some overlaps in the ownesrhips roles. As a consequence the projects efficiency ratings are also considered moderately unsatisfactory.

Finally, as regards sustainability, the ET considers that completion of the activities is likely to take place, leading to the approval of a regulatory framework, and then to the successful demonstration of technologies.

Conclusion and Recommendations

CONCLUSION 1

Maximizing Opportunities & Collaboration

Recommendation 1

Overall the Projects’ situation improved in terms of readiness but deteriorated in terms of time availability for completion

Project should take an urgent call on additional time requirements to achieve results

Contributing Conclusions Supportive Recommendations

Project stakeholders are aware of delays/time lost and share key concerns on remaining activities and time available for completion

Once an extension is obtained, work plans & activities should follow a “war footing” approach

The project should reach out to other entities such as the Stockholm Convention Centre to facilitate timely coordination

CONCLUSION 2 Consolidating success

Recommendation 2

10

Lack of fully operational M&E systems within a weak PMU contributed to operational inefficiencies

PMU needs to be strengthened and regular monitoring and evaluation systems need to be formalized and fully implemented a.s.a.p.


The project suffered numerous unforeseen delays related but not limited to policy aspects i.e. approval of Notification, and administrative aspects i.e. Environmental Clearances, etc.

Reach-out /engagement of stakeholders was limited

There were changes of stakeholder representatives and even of stakeholders

A change of NPC occurred, however no proper handover took place, which hindered the efforts of the newly appointed NPC as he did not benefit from transfer of institutional memory

Effective coordination mechanisms need to be established to ensure strong commitment from each of the project stakeholders, in their independent objectives, to catch up on operational efficiency

Greater reach-out and engagement of stakeholders needs to be actively pursued, to ensure sustainability

Country should ensure proper hand over mechanisms are effectively utilized in times of transition including full briefings by the GEF FP

-------

11

1 Evaluation objectives, methodology and process 1.1 Information on the evaluation

The Evaluation Team (ET)1 was composed of one international evaluation consultant, acting as a team leader, and one national evaluation consultant. The ET was supported by the Project Manager at UNIDO and the Project Advisor in the Regional Office, as well as by the National Project Coordinator of the Government of India. The UNIDO GEF Coordinator was briefed on the evaluation.

The purpose of the Mid Term Evaluation (MTE) is to enable the Government of India, counterparts, the GEF, UNIDO and other stakeholders to:

a) Verify prospects for development impact and sustainability, providing an analysis of the attainment of global environmental objectives, project objectives, delivery and completion of project outputs/activities, and outcomes/impacts based on indicators. The assessment includes re-examination of the relevance of the objectives and other elements of project design according to the project evaluation parameters defined in chapter V of the Terms of Reference for the MTE.

b) Enhance project relevance, effectiveness, efficiency and sustainability by proposing a set of recommendations with a view to ongoing and future activities.

c) Draw lessons of wider applicability for the replication of the experience gained in this project in other projects/countries.

The key question for this evaluation is to understand if the project has made a significant contribution to reducing the effects of POPs on human health and the environment, i.e. whether the project is going to reduce and eliminate the use and releases of PCBs to the environment through promotion of measures to minimize exposures and risks by introducing environmentally sound management and disposal of PCBs, PCB-containing equipment and, PCB-containing mineral oils and wastes, aiming at the final and virtual disposal of all PCBs inventory in India by 2025 and 2028, respectively.

1Terms of reference for the MTE of the Project

12

2 Scope and objectives of the evaluation 2.1 Information sources and availability of information

Through the documentary information and the information collected in the field, the evaluators consider that there was sufficient evidence to allow them to establish a baseline for the project.

Sources of information were sufficient to verify and document the progress and, constraints encountered during the assessment; data and information derived from interviews are qualitatively satisfactory and, this was verified through comparison of figures from different sources and through crosschecked interviews with relevant actors in an independent way, showing that respondents views and contributions were in full agreement.

2.2 Methodological remarks, limitations encountered and validity of the findings The methodology for the assessment was based on:

• A review of project documents including but not limited to: • The original project document, monitoring reports (such as progress and

financial reports to UNIDO and GEF annual Project Implementation Review reports), output reports (case studies, action plans, sub-regional strategies, etc.) and relevant correspondence;

• Reports from the Project Steering Group meetings and consultants;

• Other project-related material produced by the project.

• Interviews with the Project Stakeholders both at Headquarters and in the field; The information obtained allowed the Evaluation Team (ET) to verify that progress to date corresponds to the activities, outputs and outcomes set out in the logical framework of the project and that they are measured by the indicators defined in the logical framework.

The interviews carried out satisfactorily ensured that the views and experiences of all relevant stakeholder categories (men/women, project staff/participants, beneficiaries and non-beneficiaries, and funders) were appropriately included.

13

3 Country and project background 3.1 Country context

India, a country in South Asia is one of the most ancient civilizations, and the 7th largest in the world by area. Bounded by the Indian Ocean on the south, the Arabian Sea on the south-west, and the Bay of Bengal on the south-east, it shares land borders with Pakistan to the west; China, Nepal, and Bhutan to the north-east; and Burma and Bangladesh to the east. In the Indian Ocean, India is in the vicinity of Sri Lanka and the Maldives; in addition, India's Andaman and Nicobar Islands share a maritime border with Thailand and Indonesia.

India is a federation composed of 29 states and 7 union territories. All states, as well as the union territories of Pondicherry and the National Capital Territory of Delhi, have elected legislatures and governments. The centre, through appointed administrators, directly rules the remaining five union territories.

India: Geography at a Glance2 Capital New Delhi Geographic coordinates 20 00 N, 77 00 E Area 32,87,263 sq. km; <Land: 2,973,193 sq km, Water: 314,070 sq km> Land Boundaries 13,887 km Coastline 7,000 km Border countries Bangladesh 4,142 km, Bhutan 659 km, Burma 1,468 km, China

2,659 km, Nepal 1,770 km, Pakistan 3,190 km Climate Varies from tropical monsoon in south to temperate in north.

There are four seasons, which are recognized by the India Meteorological department. They are - Cold weather, hot weather, rainy season and the season of the retreating southwest monsoon.

Rivers The main rivers of the Himalayan group are the Indus, the Ganga and the Brahmaputra.

3.2 Socioeconomic overview

3.2.1 Social overview

With over 1.2 billion people, India is 2nd largest country in the world by population. It is the most populous democracy in the world. The four religions born in India - Hinduism, Buddhism, Jainism, and Sikhism, are followed by 25% of the world's population. Following table presents India’s key social co-ordinates.3

Population 1,236,344,631 (July 2014 est.) ~17% of the world's population Population Growth Rate 1.25 per cent (2014 est.) Birth Rate 19.89 births/1,000 population (2014 est.) Death Rate 7.35 deaths/1,000 population (2014 est.) Life Expectancy Rate 65.8 years (Males); 68.1 years (Females) in the period 2006-2011. Sex Ratio 940 females per 1000 males according to 2011 census

2 Source: https://www.cia.gov/library/publications/the-world-factbook/geos/in.html 3 Source:https://www.cia.gov/library/publications/the-world-factbook/geos/in.html

14

https://www.cia.gov/library/publications/the-world-factbook/docs/notesanddefs.html?fieldkey=2011&alphaletter=G&term=Geographic%20coordinates

http://censusindia.gov.in/

Nationality Indian Religions According to the 2001 census, out of the total population of 1,028 million

in India, Hindus constituted the majority with 80.5%, Muslims came second at 13.4%, followed by Christians, Sikhs, Buddhists, Jains, and others.

Literacy According to the provisional results of the 2011 census, the literacy rate in India stands at 74.04 per cent, 82.14% for males and 65.46% for females.

Languages There are 22 different languages that have been recognized by the Constitution of India, of which Hindi is an Official Language. Hindi 41%, Bengali 8.1%, Telugu 7.2%, Marathi 7%, Tamil 5.9%, Urdu 5%, Gujarati 4.5%, Kannada 3.7%, Malayalam 3.2%, Oriya 3.2%, Punjabi 2.8%, Assamese 1.3%, Maithili 1.2%, other 5.9% English enjoys the status of subsidiary official language but is the most important language for national, political, and commercial communication.

Despite this economic progress India suffers from a high level of poverty. In 2012, according to latest report by the Planning Commission of India (Tendulkar Committee) 21.9% of all people in India fell below the international poverty line of US$ 1.25 per day. Over the last decade, poverty has witnessed a consistent decline with the levels dropping from 37.2% in 2004-05 to 29.8% in 2009-10. The number of poor is now estimated at 250 million, of which 200 million reside in rural India.4

3.2.2 Economic profile

According to the International Monetary Fund (IMF), as of 2014, the Indian economy is nominally worth US$2.047 trillion; it is the 11th largest economy by market exchange rates, and is, at US$7.277 trillion, the 3rd largest by purchasing power parity (PPP). With its average annual GDP growth rate of 5.8% over the past two decades, and reaching 6.1% during 2011–12, India is one of the world's fastest-growing economies. However, the country ranks 140th in the world in nominal GDP per capita and 129th in GDP per capita at PPP. It is considered a newly industrialized country.

GDP - composition, by sector of origin (2013 est.):

• Agriculture: 17.3%

• Industry: 25.8%

• Services: 56.9%

Major agricultural products include rice, wheat, oilseed, cotton, jute, tea, sugarcane, and potatoes. Major industries include textiles, telecommunications, chemicals, pharmaceuticals, biotechnology, food processing, steel, transport equipment, cement, mining, petroleum, machinery, and software.

Until 1991, all Indian governments followed protectionist policies that were influenced by socialist economics. Widespread state intervention and regulation largely walled the economy off from the outside world. In 1991, India liberalized its economy and followed market based economic reforms. Since then it has slowly moved towards a free-market system by emphasizing both foreign trade and direct investment

4 Source: http://simple.wikipedia.org/wiki/Poverty_in_India

15

http://simple.wikipedia.org/wiki/Poverty_in_India

inflows. India's recent economic model is largely capitalist. India has been a member of WTO since 1 January 1995. Table below shows a snapshot of India’s external trade data5 -

Exports USD 313.2 billion (2013 est.)

Exports : commodities - Petroleum products, precious stones, machinery, iron and steel, chemicals, vehicles, apparel

Exports: partners - UAE 12.3%, US 12.2%, China 5%, Singapore 4.9%, Hong Kong 4.1% (2012 est.)

Imports USD 467.5 billion (2013 est.)

Imports: commodities - Crude oil, precious stones, machinery, fertilizer, iron and steel, chemicals

Imports: partners - China 10.7%, UAE 7.8%, Saudi Arabia 6.8%, Switzerland 6.2%, US 5.1% (2012 est.)

Economic growth was reported to be constrained by inadequate infrastructure, a cumbersome bureaucracy, labor market rigidities, regulatory and foreign investment controls, high fiscal deficits and inflation for food.

India continues to move forward with market-oriented economic reforms that include increasingly liberal foreign investment and exchange regimes, industrial decontrol, reductions in tariffs and other trade barriers, opening and modernization of the financial sector, significant adjustments in government monetary and fiscal policies, and more safeguards for intellectual property rights.

3.3 Policy and Legal framework:

3.3.1 Investment climate

After the economic reforms of 1991 and subsequent removal of regulatory and trade barriers, India emerged as a major force in the global market, becoming a hub of business process outsourcing, an attractive investment destination for foreign direct investment (FDI) and a dominant exporter of services. In a span of two decades, almost all leading multinationals have established operations in India, engaging Indian skilled labor in their business.

According to UNCTAD’s World Investment Prospects Survey 2012–2014, India is the third-most attractive destination for FDI (after China and the US) in the world. Indian markets have significant potential and offer prospects of high profitability and favorable regulatory regime for investors.

Though ranking 51st in global competitiveness, India ranks 17th in financial market sophistication, 24th in the banking sector, 44th in business sophistication, and 39th in innovation, ahead of several advanced economies, as of 2010. With 7 of the world's top 15 information technology-outsourcing companies based in India, the country is viewed as the second-most favorable outsourcing destination after the United States, as of 2009.

5 Source:https://www.cia.gov/library/publications/the-world-factbook/geos/in.html

16

3.3.2 Legal framework for environmental protection

Even before India’s independence in 1947, several environmental legislations existed but the real impetus for bringing about a well-developed framework came only after the UN Conference on the Human Environment (Stockholm, 1972). Under the influence of this declaration, the National Council for Environmental Policy and Planning within the Department of Science and Technology was set up in 1972. This Council later evolved into a full-fledged Ministry of Environment, Forest and Climate Change (MoEF&CC) in 1985 and, today is the apex administrative body in the country for regulating and ensuring environmental protection.

After the Stockholm Conference, in 1976, constitutional sanction was given to environmental concerns through the 42nd Amendment, which incorporated them into the Directive Principles of State Policy and Fundamental Rights and Duties. Since the 1970s an extensive network of environmental legislation has grown in the country.

The responsibility for prevention and control of industrial pollution is primarily executed by the Central Pollution Control Board (CPCB), at the central level, a statutory authority attached to the MoEF. The State Departments of Environment and State Pollution Control Boards are the designated agencies to perform this function at the State Level.

A policy framework has also been developed to complement the legislative provisions. The Policy Statement for Abatement of Pollution and the National Conservation Strategy and Policy Statement on Environment and Development were brought out by the MoEF in 1992, to develop and promote initiatives for the protection and improvement of the environment.

The EAP (Environmental Action Programme) was formulated in 1993 with the objective of improving environmental services and integrating environmental considerations in to development programmes.

There are three key policies relating to environmental protection in India:

• The National Forest Policy, 1988

• Policy Statement for Abatement of Pollution, 1992

• National Conservation Strategy and Policy Statement on Environment and Development, 1992

The Central government has enacted several laws for environmental protection. The Environment (Protection) Act, 1986, is the umbrella legislation. According to the Act, the term "environment" includes water, air and land and the inter- relationship which exists among and between water, air and land, and human beings, other living creatures, plants, micro-organism and property.

A comprehensive policy framework has also been developed to complement the legislative provisions, the most important being the National Environment Policy – 2006. This policy is in response to India’s national commitment to a clean environment mandated in the Constitution. It is intended to mainstream environmental concerns in all development activities and briefly describes the key environmental challenges currently and prospectively facing the country, the objectives of the policy, normative principles underlying policy action, broad indications of the legislative and institutional development needed to accomplish the strategic themes, and mechanisms for implementation and review. It also seeks to stimulate partnerships of different stakeholders i.e. public agencies, local communities, academic and scientific institutions, the investment community, and international development partners in harnessing their respective resources and strengths for environmental management.

17

http://business.gov.in/outerwin.php?id=http://www.cpcb.nic.in

http://business.gov.in/outerwin.php?id=http://envfor.nic.in/legis/legis.html%23C


http://business.gov.in/outerwin.php?id=http://envfor.nic.in/nep/nep2006e.pdf

http://business.gov.in/outerwin.php?id=http://envfor.nic.in/nep/nep2006e.pdf

3.4 Environment Overview: Major environmental issues in India are deforestation; soil erosion; overgrazing; desertification; air pollution from industrial effluents and vehicle emissions; water pollution from raw sewage and runoff of agricultural pesticides and ground water contamination. The huge and rapidly growing population is overstraining natural resources.

The Ministry of Environment, Forest and Climate Change (MoEF&CC) http://envfor.nic.in is the nodal agency in the administrative structure of the Central Government for the planning, promotion, co-ordination and oversight of the implementation of India's environmental and forestry policies and programmes. The broad objectives of the Ministry are:

• Conservation and survey of flora, fauna, forests and wildlife.

• Prevention and control of pollution.

• Afforestation and regeneration of degraded areas.

• Protection of the environment, and

• Ensuring the welfare of animals.

These objectives are well supported by a set of legislative and regulatory measures, aimed at the preservation, conservation and protection of the environment.

Besides the legislative measures, a Natural Conservation Strategy and Policy Statement on Environment and Development, 1992, National Forest Policy, 1988, a Policy Statement on Abatement of Pollution, 1992 and a National Environment Policy 2006 have also been evolved.

3.4.1 Key National Environmental regulations in India - The Environment (Protection) Act, 1986 – From time to time the Central Government prepares rules and issues notifications and guidelines for the matters under the act. The Environment (Protection) Act, 1986 (The Act), is the umbrella legislation that authorizes the Central Government to protect and improve environmental quality, control and reduce pollution from all sources, and prohibit or restrict the setting and /or operation of any industrial facility on environmental grounds. According to the Act, the term "environment" includes water, air and land and the inter- relationship which exists among and between water, air and land, and human beings, other living creatures, plants, micro-organism and property. The Hazardous Waste (HW) Rule, 1989, issued under the provision of Environmental (Protection) Act, 1986, controls and regulates the import of the hazardous wastes into the country. As per Rule 11 of the Hazardous Waste Rule, import of waste from any country to India shall not be permitted for dumping and disposal. However, the import of such wastes is allowed for processing and re-use as raw material, after the State Pollution Control Board (SPCB) has examined each case on its merit. The HW Rule was amended in 6 January 2000 to improve its applicability and implementation. The Hazardous Wastes (Management and Handling) Amendment Rules, 2003, classify used mineral oil as hazardous waste that requires proper handling and disposal. Organization will seek authorization for disposal of hazardous waste from concerned SPCBs as and when required.

18

http://envfor.nic.in/


In 2008, the Hazardous Waste (HW) Rule, 1989, was repealed and ‘Hazardous Waste (Management, Handling & Transboundary Movement) Rules, 2008 ‘ came into effect in suppression of former notifications. The Rules lay down corresponding duties of various authorities such as the MoEF&CC, CPCB, State/UT Govts., SPCBs/PCCs, DGFT, Port Authority and Custom Authority while State Pollution Control Boards/ Pollution Control Committees have been designated with wider responsibilities touching across almost every aspect of Hazardous waste generation, handing and their disposal. India is a party to the Basel Convention. It signed the Convention on 15 March 1990, ratified it on 26 June 1992, and acceded to the Convention on 22 September 1992. Ratification of this instrument represents India’s commitment to solving, in a collective manner through international cooperation, the problem of transboundary movement and disposal or dumping of dangerous and unwanted hazardous wastes. The Supreme Court Monitoring Committee on Hazardous Wastes (SCMC) has initiated return of hazardous wastes that have been wrongly imported in India. For example, it directed the return of PCB-contaminated old transformer components to Germany.

3.4.2 Environmental Concerns in INDIA

With growth in economy and industrial facilities, the Government of India reported its concern with the generation of large quantity of hazardous waste from industries, along with the hospital waste, which has been affecting public health and environment. Land degradation and loss of biodiversity were also reported of great concern. Also mainly due to increases in the number of vehicles, the air quality in cities had deteriorated and as a result there was a sharp increase in air pollution related diseases. The availability of fresh, clean drinking water was seen as a priority issue. Mitigating against the impact of climate change and improving energy security were also regarded as major challenges.

3.5 Sector-specific issues of concern

3.5.1 Indian and Persistent Organic Pollutants (PoPs) The Republic of India signed the Stockholm Convention on POPs on 14 May 2002 and ratified it on 13 January 2006. The Global Environment Facility (GEF)-funded project entitled “Development of a National Implementation Plan (NIP) in India as a First Step to Implement the Stockholm Convention on Persistent Organic Pollutants (POPs)” was approved on 14 June 2007. A National Steering Committee (NSC) was constituted under the Chairmanship of Secretary (E&F) to guide and monitor all actions needed for the preparation of the NIP, which was released in April 2011.

3.5.2 Institutions involved in POPS Management in INDIA6

Ministry of Environment, Forest and Climate Change (MoEF&CC) is the focal point in Govt. of India for all matters relating to the environment and also the Stockholm Convention on Persistent Organic Pollutants and Global Environment Facility (GEF). Various other ministries, agencies, governmental and non Governmental Institutions relevant to POPs are -

• The Central Pollution Control Board (CPCB)

6 National implementation Plan (NIP) of the Stockholm Convention on POPs April 2011

19

• The State Pollution Control boards (SPCBs)/Pollution Committee (for Union Territories)

• The Ministry of Health and Family Welfare (MHFW) • The State Health Departments • The Department of Chemicals and Petrochemicals, under Ministry of

Chemicals and Fertilizers • The Ministry of Labor • The Ministry of Foreign Affairs • The Ministry of Power • Directorate General of Foreign trade • The Central Board of Excise and Customs (CBEC) • The Ministry of Road Transport and Highways • The Ministries of Urban Development (MOUD) • The Ministry of Railways • The Ministry of Agriculture (MoA) and state departments of Agriculture

3.5.3 PCBs and India7

Poly Chlorinated Biphenyls (PCBs) were never manufactured in India and their import was banned in 1998. The requirements of PCBs for numerous applications were met through imports. There were various usages for PCBs in India, though the power transformers were the major consumer of PCBs as insulating fluid. As per the national implementation plan (NIP) of the Government of India, the current stock of PCB-containing oil is estimated at 10,000 tons.

Sector wise distribution of PCBs in India –

Sector No. of Transformers

% weight of PCB containing oils

Weight of PCB containing oils (tons) including retro

filling Power Sector 407 71.3 7016.034 Steel Sector 913 18.0 1772.428 Cement Sector 34 0.5 49.290 Fertilizers 16 0.3 28.680 Others 178 9.9 971.230

Total 1548 100 9837.662

Select PCB contaminated sites in India are:

• Bhilai Steel plant, Chhattisgarh • Rourkela Steel Plant, Orissa • Neyveli Lignite Corporation limited, Tamil Nadu • IISCO Steel Plant, West Bengal • Dhulekote storage and disposal facility, Haryana • Panki Thermal Power plant, Uttar Pradesh • Ship Breaking Yard, Alang, Gujarat

It is estimated that a total number of about 45,000 power and about 3,500,000 distribution transformers exist. While the total amount of imported PCB-oils as well as

7 National implementation Plan (NIP) of the Stockholm Convention on POPs April 2011

20

PCB-containing equipment is unknown, the estimated quantity of pure PCB-oils and PCB containing equipment in India according to the initial inventory amounts to 7,700 tons that is targeted for safe disposal in this Project.

3.5.4 Priorities and Action plan strategies

Based on the consultative NIP development process, the Government of India identified following priorities for the implementation of NIP:

• Environmentally Sound Management and Final Disposal of PCBs

• Environmentally Sound Management of Medical Wastes

• Development and promotion of non POPs alternatives to DDT

• Implementation of the Best Available Technology (BAT)/ Best Environmental Practices (BEP) strategies for elimination / reduction of unintentional POPs emissions of the priority industry sectors identified in the NIP of India

• Management of PVC plastic waste to avoid incineration / dumping the landfill for preventing releases of Dioxins and Furans due to burning

• Capacity building, demonstration of production and promotion of bio-botanical neem derived bio-pesticides as viable, eco-friendly, bio-degradable alternatives to POPs pesticides

• Identification of sites contaminated by POPs chemicals and of remediation process at the potential hotspots

• POPs and pesticides management in India

• Inventorization of newly listed POPs

• National POPs monitoring India program and

• Strengthening institutions and capacity building for effective and efficient implementation of the NIP in India.

The NIP implementation has been harmonized with the 5-year planning process in India and with the 5-year replenishment of the GEF. In this regard the first NIP is planned until 2022. The 5-year planning seeks to provide guidance to development policies and programmes that promote sustainable management of the nation’s resources to attain the ultimate objective of sustainable development. The NIP implementation will also be integrated within the 5 year plans of the Government of India.

----

21

4 Project summary 4.1 Project Fact Sheet

Country INDIA

Project title Environmentally Sound Management and Final Disposal of PCBs in India

GEFSEC (PMIS) ID 3775 - Full Size Project (FSP)

GEF Agency Project ID GF/IND/10/001

GEF Focal Area and Operational Program

GEF-4, Persistent Organic Pollutants POPs SP-1, SP-2

Agency UNIDO

Other Executing Partners Ministry of Environment, Forest and Climate Change (MoEF&CC) – Ministry of Power (MOP)/Central Power Research Institute (CPRI)

Project Approval Date 30 December 2009

Date of Project Effectiveness 13 January 2010

Total project Cost $ 43,450,000 (excluding support costs)

GEF Grant $14,100,000 excluding PPG

Agency Fee $ 1,445,000 as Support Costs (10%)

GEF Project Preparation Grant Amount

$350,000

4.2 Brief description The project focuses on the reduction and elimination of PCBs in the power sector through (a) development of appropriate legislation, (b) provision of capacity building for key stakeholders, (c) development of an ESM system for PCBs, PCB-containing electrical equipment and waste and incorporating it into a national policy framework, (d) gradual phase- out of PCB-containing equipment (transformers and capacitors), (e) disposal of all PCBs, PCB-containing equipment and wastes, (f) strengthening environmental monitoring capacities and (g) identifying the most appropriate mitigation measures to reduce social costs of complying with the Stockholm Convention.

22

In addition, the project aims to start creating a national inventory of non-electrical equipment and other articles containing more than 0.005 percent of PCBs as required by the Stockholm Convention 8.

4.2.1 Project Objectives

The overall objective of the project is to reduce and eliminate the use and releases of PCBs to the environment through promotion of measures to minimize exposures and risks by introducing environmentally sound management and disposal of PCBs, PCB-containing equipment and PCB-containing mineral oils and wastes aiming at the final and virtual disposal of all PCBs inventory in India by 2025 and 2028, respectively.

The immediate objectives of the project are to:

• Strengthen the legal and regulatory framework for environmentally sound management (ESM) and disposal of PCBs, PCB-containing equipment and PCB-containing mineral oils and wastes;

• Improve institutional capacity at all levels of PCBs disposal management; Removal of 7,700 tones of PCBs, PCB-containing equipment and PCB-containing mineral oils and wastes from targeted sites and transport them to disposal unit; and

• Dispose of 7,700 tones PCBs, PCB-containing equipment and PCB-containing mineral oils and wastes in an environmentally sound manner.

4.2.2 Project Outcomes and Outputs

The project consists of 5 substantive Outcomes, each with specific outputs, as described below:

Outcome 1 Strengthened policy and regulatory framework to comply with the obligations under the Stockholm Convention

Outcome 2 Relevant institutions in India are enabled to manage PCBs in an environmentally sound manner, as well as awareness-raising on the adverse effects of PCBs

Outcome 3 Targeted regional implementation for ESM of PCBs, PCB-containing equipment and waste

Outcome 4 Regional capability for final treatment and disposal of PCBs, PCB-containing equipment and wastes

Outcome 5 Project Management and monitoring and evaluation

The table below presents the key outputs and expected outcomes -

8 FSP Prodoc 25 November 2009 revision

23

Project Outputs Expected Outcomes

Output 1.1 Legal and regulatory framework for the ESM of PCBs reviewed and assessed

Output 1.2 Legal and regulatory framework at the national level established or upgraded

Output 1.3 National legal and regulatory framework implemented in targeted pilot states

Output 1.4 Pollution prevention and management of PCBs, PCB-containing equipment and waste in consonance with ESM guidelines

This will result in a stronger legal and regulatory framework for environmentally sound management (ESM) and disposal of PCBs, PCB-containing equipment and wastes. The Government of India enacted the laws to ban the import of PCBs and regulate the handling of PCBs as a hazardous waste in line with requirements of the Basel Convention. However legislation has not been promulgated to regulate the labeling requirements of the SC. The project will work with legislative and regulatory agencies to update existing laws and regulations and, where necessary, develop new regulatory approaches to specifically address PCBs management.

Output 2.1 Institutional capacity for ESM of PCBs, PCB-containing equipment and wastes evaluated

Output 2.2 Training workshops for key stakeholder undertaken

Output 2.3 A national tracking and record keeping system (PCB inventory database) established and maintained countrywide (29 states and 7 union territories)

Output 2.4 Sampling, analysis and monitoring capacity evaluated and strengthened in 13states

Output 2.5 Awareness raising carried out

This will result in improved institutional capacity at all levels of PCBs, PCB-containing equipment and waste disposal management. PCBs waste generators and contaminated site owners are often unaware of the potential environmental, legal, and human health risks that their current and past operations entail. In addition, waste generators and owners lack the managerial and technical capacity to manage POPs wastes in an environmental sound manner.

Output 3.1: Dedicated environmentally sound maintenance capacity for PCBs, PCB containing equipment and wastes established

This will increase capacity of PCB owners in India to properly manage PCBs, PCB-containing equipment and wastes.

Output 4.1: Management system for identification, tracking, collection, packaging, transport, interim storage, record keeping, and disposal of PCBs, PCB-containing equipment and waste developed and operational in 13 states

Output 4.2 ESM and transport to interim storage sites of PCB-containing materials carried out including specialized transport vehicles for highly concentrated PCBs with GPS and adequate preparedness measures in case of emergency on transport routes to the stationary disposal unit

Output 4.3 Final ESM treatment of at least 7,700 tons of PCBs, PCB-containing equipment and PCB-contaminated oil and wastes undertaken

This will establish a regional capacity to treat and dispose of PCBs, PCB-containing equipment and wastes.

Output 5.1 Project management structure established

This will provide ongoing project management, monitoring, and evaluation, including establishment of a Project Steering Committee

24

4.3 Project implementation UNIDO is the GEF implementing Agency (IA) for the project. The Ministry of Environment, Forest and Climate Change (MoEF&CC) is the government co-coordinating agency for the project. In addition, two key stakeholder agencies play crucial roles in the project. The first is the Ministry of Power (MOP), the counterpart agency, which is responsible for assisting in the implementation of the activities and measures for limitation, elimination and monitoring of import and use of PCB containing equipment and reduction of unintentional production of POPs chemicals. The other key stakeholder is the Central Power Research Institute (CPRI), which is the national executing agency/cooperating agency for the project. The MoEF&CC entrusted CPRI to be the National Executing Agency of this project. CPRI under the Ministry of Power is a government agency and was at the time of approval of the project selected to execute the PCB-related activities of the NIP as well as post-NIP activities due to their vast experience in environmental sound management (ESM) of PCBs oil and other PCB- containing hazardous materials. The Central Pollution control Board (CPCB), under MoEF&CC is also involved in the project implementation activities. CPRI is responsible for the following issues:

• Coordinate legislative activities while recommending amendments and additions to relevant legislation and regulations as well as develop guidelines for PCBs related activities;

• Facilitate cooperation and coordination between stakeholders and provide the stakeholders with management tools;

• Conduct inventories of production and utilization of PCB-containing equipment and PCB-containing oils and wastes;

• Establish a database and an information sharing network;

• Provide individuals, agencies and companies with PCB-related information;

• Organize capacity building activities such as trainings, workshops and seminars; and

• Monitor and assess the implementation of responsibilities and duties of stakeholders and regularly report to relevant governmental authorities.

UNIDO is responsible for the supervision of project management, facilitate stakeholder co-ordination, monitoring of the project, and reporting on project performance to the GEF. UNIDO is in charge of procuring the international expertise and equipments needed to deliver the outputs planned under the project components.

Output 5.2: An M&E mechanism designed and implemented according to GEF M&E procedures

composed of national and local stakeholder agencies, establishment and staffing of the project management team at the national and local levels, recruitment of national and international consultants, execution of a management training program for project staff (particularly at the local level), and ongoing monitoring and reporting of project activities.

25

Steel Authority of India’s (SAIL) Bhilai Steel Plant (BSP) also has a key role in the project implementation. SAIL, Under Ministry of Steel, joined the project in its very 1st year. SAIL as lead beneficiary of this project will host the project implementation site at BSP, with a co-financing support estimated over Rs. 140 Crore (~ USD 23.3 mn) in the form of cost of land, civil structure, raw materials, utilities, required manpower and operating expenses for the facility.

4.4 Positioning of the UNIDO project There is a need for development and extensive targeted capacity building enhancing the decision-making, managerial, and technical capabilities of government officials to implement the NIP provisions and provide guidance to public and private enterprises in environmentally sound PCB management and other priorities. NIP implementation at the country level requires the development of adequate capacities at central and local levels. The number of qualified human resources for the enforcement of the existing and future regulations avoiding non-compliance of legislations and improper management of PCBs should also be enlarged. Human and technical capacities for PCBs monitoring, especially the proper laboratory services for PCBs analysis has to be strengthened as well. The introduction of internationally accepted analytical testing methodology, national standards and accreditation procedures shall also be strengthened.

Legislation and regulations with regard to hazardous waste are in place in India; however, there are serious shortcomings in implementing and enforcing these regulations in context with PCB management in the country. It is partly because the PCBs wastes have been accumulating as PCB-containing equipment has become obsolete and their repair would not be feasible any more and partly because India has to meet its obligation related to BAT/BEP in disposal of PCBs. In addition the old equipment leaks and continuously contaminates their sites posing health and environmental hazards.

Environmentally unsound storage conditions at these sites entail high risk of PCBs release, with concomitantly high risk of PCBs contamination of soil and ground water. Capacity does not currently exist to dispose of obsolete PCB-containing equipment, oil and wastes in an environmentally sound manner.

The project addresses national priorities such as to improve legislation on POPs chemicals, to eliminate PCB-containing equipment, to reduce PCBs releases from industrial wastes and sewages, to improve environmental performance in power sector, to improve environmental performance in industry sector, and to identify PCBs wastes and contaminated sites and their environmentally sound and safe management.

This project will strengthen the management and disposal of PCBs, PCB-containing equipment and wastes in an environmentally sound manner and eliminate the risk of PCBs to human health and the environment in India, in the South Asia region and globally. Methodologies for PCBs waste site risk assessment, handling, clearance, collection, labeling, packaging, transportation, interim storage, disposal, and emergency response procedures will be developed according to strict internationally accepted technical standards and guidelines including those developed by COP to the Basel Convention. The emphasis will be put on the cost-effectiveness and sustainability of the proposed measures.

The project will enable the Government of India to duly report on the progress in eliminating PCBs pursuant to Annex A, Part II, subparagraph (g) of the Convention according to Part C of the format reporting under Article 15 of the Stockholm Convention (Annex to Decision SC- 2/18 of the Conference of the Parties).

26

4.5 Counterpart organization(s) Ministry of Environment, Forest and Climate Change (MoEF&CC), Government of India

Ministry of Power (MOP)/Central Research Power Institute (CPRI)

----

27

5 Project assessment 5.1 Design

• Is the project’s design adequate to address the problems at hand? In particular, further analyze the proposed use of dechlorination as the sole PCB decontamination method. Was a participatory project identification process applied and was it instrumental in selecting problem areas and national counterparts?

Overall the project as designed and implemented is considered to be an adequate vehicle to assist the government to address the set objectives. In particular addressing policy issues required to ensure compliance, but also to facilitate understanding of the issues at hand and, engage institutions in an early dialogue, is considered a key to the success of the proposed approach.

The project adequately addresses short and long-term challenges and follows clear thematically focused development objectives, all of which are elements for success. In addition the evaluation team (ET) was able to document the fact that a strong participatory process was followed, starting with the development of the NIP, and this was supported by strong leadership from the GoI through its MOEF 9.

However the ET found that participation and involvement of the private sector and of civil society were initially lacking, which is critical from the scalability perspective of the project. Also, while the project took an initial stock of various “environmentally sound technologies for PCB disposal” it did not carry out an assessment of the technological readiness of the country - mainly in terms of availability and affordability of technological options within the country 10 - which is a key component from the policy development and sustainability perspective.

Finally, the design of the project most importantly did not properly factor in the processes/time required to develop and establish the regulatory framework and although this could have compromised the project – and indeed came close to doing so as is documented below - the ET considers it likely to highly likely that the project will deliver on results.

• Does the project have a clear thematically focused development objective, the attainment of which can be determined by a set of verifiable indicators? Was the project formulated based on the logical framework approach?

Yes, the project was formulated based on the logical framework approach. The narrative synthesis is consistent and the products are necessary to achieve the expected results. The baselines and targets are clear; the indicators are suitable; the verification sources are accessible, and the risks and assumptions identified are external critical factors that are beyond the control of the project.

• Was the project formulated with the participation of national counterpart and/or target beneficiaries? Were relevant country representatives (from government, industries and civil society) appropriately involved and

9 Interview data 10 For example non combustion technologies (dechlorination) contemplated in the ProDoc had to be complemented with combustion technologies once the estimates of stockpiles of pure PCB became known

28

participating in the identification of critical problem areas and the development of technical cooperation strategies?

The project was formulated with the participation of national counterpart, but the ET did not receive evidence regarding organization of a discussion for target beneficiaries, the outcome of which would have been used to craft a common vision document. MOEF and CPRI were involved since the time of the preparation of the NIP, and participated in the project formulation. The direct involvement of Industry and Civil society however is not evident to the ET.

There should have been a larger participation of industry (private sector) and civil society in particular, as the later is and will be the first to be directly affected by PCB hazards.

5.2 Relevance Relevance was assessed by the Evaluation Team (ET) as being Highly Satisfactory, as detailed below.

----

• Relevance to national development and environmental priorities and strategies of the Government and population of India and, regional and international agreements. Relevance to target groups: relevance of the project’s objectives, outcomes and outputs to the different target groups of the interventions (e.g. companies, civil society, beneficiaries of capacity building and training, etc.). Relevance to the GEF and UNIDO: Are the project’s outcomes consistent with the focal areas/operational program strategies of GEF? Ascertain the likely nature and significance of the contribution of the project outcomes to the wider portfolio of the GEF Operational Programme (OP) #14 (or CHEM-1). Are they in line with the UNIDO mandate, objectives and outcomes defined in the Programme & Budget and core competencies?

Although Polychlorinated Biphenyls (PCBs) were never produced in India and import was banned in 1998, India, through the MOEF, selected Environmentally Sound Management (ESM) and disposal of PCBs as one of the first action priorities of its post-NIP program. The decision to prioritize this sector was influenced by the tight SC implementation timeframe, as well as by the information that was provided by the initial survey and inventory carried out by the Central Power Research Institute (CPRI) from 2004 to 2008 11. This inventory estimated that 45,000 power and 3,500,000-distribution transformers were to be found in the 29 states and 7 union territories of India. While the total amount of imported PCB-oils as well as PCB-containing equipment is unknown, the estimated quantity of pure PCB-oils and PCB-containing equipment, according to the initial inventory, amounted to 7,700 tons. Given the above, the project is considered relevant to the national priorities as it addresses a set of pertinent issues including improvement of the legislation on POPs chemicals, elimination of PCB-containing equipment, reduction of PCBs releases from industrial wastes and sewage, improvement of environmental performance in the power sector, improvement of environmental performance in the industry sector, and

11 This initial survey included establishment of a preliminary inventory on PCB-containing electrical equipment as well as the evaluation of India’s capacity for ESM and disposal

29

identification of PCBs wastes and contaminated sites and their environmentally sound and safe management. The ultimate outcome of the project, which seeks to dispose of an estimated (but already considerable) amount of PCBs, is clearly aligned with the overarching environmental priorities of the national government, as well as those of the industrial sector. Ultimately this is also considered to be in the best interest of the population, as a whole.

The relevance of the project to the target groups is also clear and interviews provided ample evidence that these groups had not only been reached, but demonstrated a good to very good understanding of the issues at hand. The project is considered relevant to the GEF operational programme and UNIDOs thematic priorities through the application of ESM in the management and disposal of PCB-containing equipment, oil and wastes. In particular as pursuant to POPs focal area Strategy and Strategic Programming 1 and 2 (GEF-4 2007-2010), the GEF was to finance projects aiming at strengthening capacities for NIP implementation in order to assist countries to meet their obligations under the SC. This includes POPs reduction measures, and partnering in investments in NIP implementation that:

a) Sustainably reduce POPs production, use and releases through phase-out, destruction in an environmentally sound manner, and

b) Use substitute products and alternative practices that lead to reduced environmental and health risks caused by POPs.

As detailed in the Prodoc and concluding a series of clearly stated arguments in the relevant section, the final justification for GEF assistance sets forth that not only the objectives, outcomes and outputs of the project meet the goals and objectives of GEF SP-1 and 2 in the POPs focal area for GEF-4, but also that outcomes are in line with requirements of the SC and follow Basel Convention Technical Guidelines; that the project applies ESM and BAT/BEP in management and disposal of PCBs, PCB containing equipment and wastes posing a major public health and environmental threat; and finally that the project will ensure the sustainability and replicability of its outputs, significantly increasing global benefits.

• Does the project remain relevant taking into account the changing

environment? Is there a need to reformulate the project design and the project results framework given changes in the country and operational context?

The project has remained relevant; in particular as it has a low direct connectivity with the changing environment. Given no major change in the overall project environment is evidenced yet, as in particular policies are not in place, and although this factor could contribute to maintaining a “low pressure level” to act as old transformers are still operational (why replace what is not broken? etc.), the ET does not consider this to have affected relevance.

Furthermore this relevance should increase in the near term, as the project achieves its intended outputs and approaches eventual outcomes/impacts. Also the project would become more relevant with the enactment of legislative notification by the Government of India on the Safe Disposal of PCBs in coming years, preferably during the project tenure.

The ET does not consider that there is a need to redesign the project at this late stage, however efforts should be made to strengthen awareness raising activities for the private sector so that when policy is in place, all stakeholders will be adequately informed and ready to participate.

30

5.3 Effectiveness The Effectiveness of the project was assessed against the outcomes, as stated in the project document, and effectiveness has been determined by the ET to be Moderately Unsatisfactory, as detailed below.

Overall the starting phase of the project consumed more time than was estimated due to a series of unforeseen events and overlaps in the ownership roles. This required re-working and re organizing the stakeholders engagement as well as the control and co-financing commitments. In addition to this, a series of unforeseeable events further hindered the effective start of the project. As regards the latter, the ET documented the following:

• No Inception Workshop organized. This in spite of numerous attempts from the Field Office. This is considered to have seriously hindered implementation of the project.

• Change of NPC, with no proper handover process. This hindered the efforts of the newly appointed NPC who did not benefit from transfer of institutional memory;

• Change of operating entity. Further to a change in leadership/priorities at CPRI 12, the Bilai Steel Plant (BSP) was approached to partner in the project as hosting facility of one of the pilot projects. This was a time consuming process that started in late 2010 with the approval in principle by the TWG for MOEF/UNIDO to enter into discussions with SAIL on the cost of utilities and the modus operandi for running the Operating Unit 13. The final approval was granted by the Board of Directors of SAIL on 13 September 2011;

• 1st tendering process (Feb 2011) failed. The technical envelope was cleared in May 2011, however the commercial envelope far exceeded the available budget;

• Fund-flow arrangement decision of the Department of Economic Affairs (DEA) of the GoI (September 2011). This decision to route all funding for all projects (including GEF funded projects) hampered project implementation. Numerous time consuming interventions at different levels contributed to official Clearance of the issue by the DEA in October 2012;

• 2nd tendering process (Oct 2011) partially succeeded. This required new ToRs as process was split into two parts: a) blends and b) pure PCBs. Technical review of Part a) was cleared in November/December 2011 however the process failed for financial reasons (commercial proposal outside of the envelope), and had to be reissued. Although for Part b) a Ramky (with Kinetrics) was successfully approved by the TWG, negotiations were necessary to clarify outstanding points between Ramky and the BSP. The contract was finally signed on 24 May 2012;

• 3rd tendering process. ToRs for part a) prepared end 2012 and presented to TWG on 16 February 2013, which requested a second opinion on the

12 CPRI remains a part of the project as Executing Agency for soft-activities (non investment, ie workshops, publications, etc.) and it is expected it will coordinate deployment of the mobile facility 13 Minutes of the TWG meeting held on 18 November 2010, which further stipulate that once SAIL agreed to taking on responsibilities, MOU could be signed by MOEF and SAIL

31

destruction technology being recommended. Authorization to publish RFP granted in July of 2013 and technical and commercial evaluation took place February/March 2014. Price negotiations with vendor took place during May 2014 and contract was awareded in September of 2014. At the time of the MTE the counter signature of the vendor was still pending;

• Finally, further to the first visit of Ramky to the BSP site, negotiations were undertaken to clarify and resolve issues regarding the now necessary Environmental Clearance for construction of the unit 14. The process started in mid 2012 and the final administrative permits (Consent for Establishment) were approved and provided to BSP in March 2014.

Notwithstanding the above, the project has quite remarkably managed to hold the course and significant progress has been accomplished to date, as is detailed below. The results achieved at various levels provide a well-laid foundation for taking off towards the qualitative achievement of key project outcomes, in the next phase.

This in turn will likely improve the prospects of achieving the long-term objectives of the project.

• What outputs and outcomes has the project achieved so far (both qualitative and quantitative results)?

-------

Outcome 1: Strengthened policy and regulatory framework to comply with the obligations under the Stockholm Convention Output 1.1 Legal and regulatory framework for the ESM of PCBs reviewed and assessed

This output includes 4 activities: the evaluation of the existing national and regulatory framework; Preparation of a gap analysis between the Stockholm Convention and the existing legal and regulatory framework; Recommendations to legislative bodies for new/revised laws to implement the requirements of the Stockholm Convention; Recommendations to regulatory bodies for new or revised regulations and guidelines to implement the requirements of the Stockholm Convention.

At the time of the MTE, the first two activities had been completed and both the evaluation and the gap analysis reports were provided to the ET.

As regards the third and fourth activities, Recommendations have been transmitted to the relevant legal department of the government and, final approval of the Notification is expected further to Parliament approval (within the next 6 to 10 months). This relatively short timeframe for approval appears to be a direct result of the recent change in government, which reportedly has established a more efficient and shorter approval system that could benefit the project.

Output 1.2 Legal and regulatory framework at the national level established or upgraded

14 Initially the unit was going to be housed within the walls of the existing BSP. However, the ET was informed that due to delays, and to BSPs own growth plans, the decision was taken to commission the unit outside the walls of the existing plant. This in turn triggered a very lengthy administrative process to obtain an Environmental Clearance for the construction of the unit.

32

This output includes 2 activities: Issuance of new and/or revised regulations to implement Stockholm Convention requirements; Enactment of new and/or revised laws to implement Stockholm Convention requirements.

Pending, as completion of both of these activities is dependent on completion of the activities of output 1.1, in particular the Draft Notification approval process.

Output 1.3 National legal and regulatory framework implemented in targeted pilot states

This output includes 3 activities: Evaluation of existing State enforcement of PCB management related laws and regulations; Identification of gaps between State implementation and National and Stockholm Convention requirements; Support for State adoption of revised and/or new measures to ensure environmentally safe management and disposal of PCB contaminants.

The first two activities have been completed and reports were made available to the ET.

Activity regarding provision of support to State is to be carried out once the Law is in place.

Output 1.4 Pollution prevention and management of PCBs, PCB-containing equipment and waste in consonance with ESM guidelines

This output includes 3 activities: Evaluate current practices for management of PCBs, PCB- containing equipment and wastes; Develop guidelines for management of PCBs, PCB-containing equipment and wastes in consonance with ESM guidelines; Stakeholder and PCB owners training in the management of PCBs, PCB-containing equipment and wastes in consonance with ESM guidelines.

The first activity was completed by CPRI at the time of the update of the evaluation prepared for the NIP and the report was made available to the ET.

The second activity saw the completion of the guidelines, which are available both in printed and electronic, form. Over 1,000 copies of these were printed and distributed. Both of these activities were co-financed by CPRI.

As regards the training, at the time of the MTE, 6 out of the 10 planned workshops had been completed nationwide and over 1,000 participants trained. UNIDO was present in 4 out of the 6 workshops and provided input, backstopping and support to preparation of manuals and to the workshops themselves. In total 2 workshops are planned each in the North and South; 1 in the East; and 6 in Central India.

As well, the ESM guidelines were disseminated to PCB owners who are reportedly satisfied with results and consider the drafts to be “good” 15.

Outcome 2: Relevant institutions in India are enabled to manage PCBs in an environmentally sound manner as well as awareness raising on the adverse effects of PCBs Output 2.1 Institutional capacity for ESM of PCBs, PCB-containing equipment and wastes evaluated

This output includes 3 activities: Identification of stakeholders to be targeted in institutional capacity building efforts; Evaluation of current stakeholder institutional

15 Interview data

33

capacity for ESM of PCBs, PCB-containing equipment and wastes; Identification of stakeholder capacity building needs.

All activities have been satisfactorily completed as a result of the update of the NIP, by CPRI, and outcomes of the field visits have been appropriately documented (reports, minutes, etc.).

Output 2.2 Training workshops for key stakeholder undertaken

This output includes 5 activities: Training materials on the planning and organization of PCB phase-out and disposal methods developed; Workshop for planning and organization of PCB phase-out and treatment methods developed; Training workshops for PCB owners; Capacity building for workplace safety monitoring by safety inspection agencies; Capacity building for environmental inspection agencies.

The first two activities on treatment methods and technologies to be adopted are reported as having been completed. The materials have been printed and distributed “in the thousands”, including e-copies, to PCB owners and stakeholders. The workshops have been conducted as described above and the materials prepared for delivery of trainings required between 8 to 10 hours to be fully presented and discussed.

Output 2.3 A national tracking and record keeping system (PCB inventory database) established and maintained countrywide (29 states and 7 union territories)

This output includes 9 activities: PCB owner workshop to introduce PCB reporting requirements; PCB owner capacity building to identify and label PCB- containing equipment; Provision of inventory monitoring kits and other monitoring supplies; Inventory survey of PCB use in power sector; Inventory survey of PCB use in ship breaking sector; Inventory survey of PCB use in other non-power sectors; Inspection and verification; Initial inventory completion; Inventory updates.

All of the above are reported as completed or ongoing and it is expected that during Q2/Q3 of 2015, the information gathered for the activities above will have been integrated in a database. The dialogue for the development of this database is also reported as ongoing. The ET was also informed of visits and sampling missions made to shipbreaking sites.

Output 2.4 Sampling, analysis and monitoring capacity evaluated and strengthened in 13 states

This output includes 2 activities: Upgrade laboratory facilities to monitor PCBs; Technical training in PCB monitoring

The laboratory was upgraded at CPRI during the NIP, and updated a second time under this UNIDO subcontract for the GEF. Technical training has been provided regularly for laboratory staff by the institution, through regular and documented training updates. Sampling is being re-conducted, as BSP required further confirmation of PCB contents in transformers.

Output 2.5 Awareness raising carried out

This output includes 5 activities: Information on PCB risks and risk minimization disseminated through print media; Information on PCB risks and risk minimization disseminated through online sources; Information on PCB risks and risk minimization disseminated through televised public service announcements; Public awareness raising through targeted workshops; Policy makers awareness building.

34

The activities above have all been reported as completed and, for the workshops participation was secured for a broad audience that included NGOs. Additionally awareness raising workshops for the ship breaking (dismantling) sector were arranged by CPRI with a particular focus on PCB containing components, particularly those destined for recovery/recycling.

Outcome 3: Targeted regional implementation for ESM of PCBs, PCB-containing equipment and waste Output 3.1: Dedicated environmentally sound maintenance capacity for PCBs, PCB containing equipment and wastes established

This output includes 4 activities: Identification of technical and technological needs to implement EMS for PCBs, PCB containing equipment and wastes; Dedicated maintenance facility design, certification and procurement; PCB owner maintenance facility ESM upgrades; Technical training of stakeholders as well as interested candidates from other countries.

This is reported as completed in particular through a “Vendors Workshop” organized by UNIDO in August of 2010, with support from the MOEF, and CPRI (Chair of the TWG). In addition elements of the activities above have been used in the manuals.

Outcome 4: Regional capability for final treatment and disposal of PCBs, PCB-containing equipment and wastes Output 4.1: Management system for identification, tracking, collection, packaging, transport, interim storage, record keeping, and disposal of PCBs, PCB-containing equipment and waste developed and operational in 13 states

This output includes 6 activities: Develop guidelines for PCBs, PCB –containing equipment and waste identification, tracking, and record keeping; Develop guidelines for PCBs, PCB –containing equipment and waste collection, packaging, and transportation; Develop guidelines for PCBs, PCB –containing equipment and waste interim storage; Develop guidelines for PCBs, PCB –containing equipment and waste disposal; Develop information management software for PCB management system; Train stakeholders in management system requirements and procedures.

The first four activities were completed and 4 sets of guidelines were prepared. This was done with the support of a UNIDO expert and the MOEF, with final approval and adoption by CPRI. These guidelines are reproduced as a part of the manuals and as such have been broadly disseminated (see above).

Output 4.2 ESM and transport to interim storage sites of PCB-containing materials carried out including specialized transport vehicles for highly concentrated PCBs with GPS and adequate preparedness measures in case of emergency on transport routes to the stationary disposal unit

This output includes 3 activities: Identify locations for interim storage facilities; Interim storage facility design, construction, provisioning, and commissioning; staffing and staff training for interim storage facilities.

The first activity has been completed, based on the inventory developed during the NIP and an updated under this project. This covered identification of stocks and information was also provided on proper storage methods, within the available infrastructures.

Ramky has completed the design of the storage facility for BSP, and the staff training is ongoing.

Output 4.3 Final ESM treatment of at least 7,700 tons of PCBs, PCB-containing equipment and PCB-contaminated oil and wastes undertaken

35

This output includes 7 activities: Technology requirements specification and procurement of decontamination equipment; Decontamination of PCB-contaminated oil; Identify location for stationary treatment facilities; Treatment facility design, technology requirements specification and procurement; Treatment facility construction, equipping and commissioning; Treatment facility staffing and staff training; Disposal of PCBs, PCB-containing equipment, PCB contaminated oil and other PCB -containing wastes.

This output includes activities that account for approximately US $12 million of the overall budget and is broken down into approximately US$4 million for the mobile facility to treat pure PCB (a)), approximately US$ 4.3 million for the treatment facility for PCB contaminated oils (b)), and the Mobile facility, for which the tendering process is still ongoing 16. At this stage the tender process has been completed and the orders are placed, however this process ran into a series of unforeseen complications as was detailed above, i.e. for the 1st tender process the financial envelope was too high; for the 2nd tender process, the financial envelope for part a) was too high; for the 3rd tender process only one bidder succeeded in his application and this required preparation of a “sole source justification” waiver process (won by Ramky/plascon (plasma)); finally, the 2nd bidding process for part b) was cleared (won by Ramky/Kinetricks (dechlorination)). It is important to note that tax exemptions will be granted for the equipment components above.

The treatment plant (Part b)) will be housed adjacent to the main premises of the BSP facility in Chhattisgarh State and the ET was informed that the design process is completed (submitted and approved 17 ). As well, requirements are completed, including construction of elements for the process such as vessels, piping etc.

The construction of the facility is now dependent on the consent of the Management Board of BSP, expected close to end of 2014. It is important to note that this needs to be considered urgent (and presents a high risk to the project) as the ET was informed that given the delays incurred in securing the administrative permits for the construction of the facility, Ramky has incurred costs (legal proceedings required to obtain a Stay from the court) as the supplier of the plant was considering the cancellation of the contract. This stay is limited in time.

Part a) will initiate as soon as the contract is signed and the ET was informed that this is in the final stages of approval.

Outcome 5: Project Management and monitoring and evaluation Output 5.1 Project management structure established

This output includes 6 activities: Establish POPs Management Unit (PMU) and appoint project leadership staff; Establish Project Steering Committee (PSC); Recruit project advisor(s), policy experts, and technical experts in PCBs management, project evaluation, and program development; Hold project management training for project management staff; Establish PMUs within participating organizations and sign project participation contracts; Establish project management information system (MIS), including a project website to disseminate information to stakeholders.

16 ToRs for the Mobile facility where drafted in July 2014 and approved by the TWG Chair in September 2014. 17 Including 3D drawings, detailed engineering design, etc.

36

The PMU is in place at MoEF 18 and is led by the National Project Coordinator (NPC) a high-level government representative, assisted by other government personnel and consultants. At the time of the MTE, 2 project advisors were on board and although their contracts were coming to an end, assurances were provided that it was planned to extend these and/or recruit additional consultants to ensure project completion.

The Technical Committee (TC aka Technical Working Group (TWG)) 19 is in place and is composed of senior representatives of all stakeholders (CPRI, MOP, Ministry of Steel, SAIL, UNIDO, Power Grid Corporation of India Limited, MOEF, and UNIDO).

The Project Steering Committee (PSC) as a whole have met formally once, however stakeholders do not appear to see this as a hindrance as informal bilateral meetings are arranged on an as needed basis to resolve outstanding issues.

Stakeholders have established PMUs in line with UNIDO work plan for example with CPRI and BSP. Reportedly the members of these unites are clearly identified and meet regularly.

Although to date only a small MIS system developed during NIP is available, the ET received assurances that this would be updated to cover the needs under the present project.

Output 5.2: An M&E mechanism designed and implemented according to GEF M&E procedure

This output includes 9 activities: Prepare and hold Inception Workshop; Measure impact indicators; Carry out annual project financial audits; Prepare Annual Project Reports and Project Implementation Reviews; Hold annual Tripartite Review meetings; Carry out mid-term external evaluation; Prepare and hold Project Completion workshop; Carry out final external evaluation; Complete project Terminal Report.

The ET confirmed that an Inception Workshop (IW) for the project did not take place, and this although it is clearly described and budgeted for in the Project Document 20. Unfortunately the ET was not able to document justification for this decision and considers that this single non-action very likely substantially hindered effective project implementation. Indeed, to quote the project document – one of the essential uses of an IW is [to] provide an opportunity for all parties to understand their roles, functions, and responsibilities within the project's decision-making structures. It follows that if this does not take place, at best a project is off to a confusing start with no clear guidelines and proper distribution of roles and responsibilities.

• Has the project generated any results that could lead to changes of the assisted institutions? Have there been any unplanned effects?

Yes, to a reasonable extent the project has generated positive results in terms of assisting and building leadership capability (CPRI and SAIL), and/or as a result of capacity building activities for other stakeholders. It is foreseen that major results in

18 MOEF is the GEF Operational Focal Point (FP), the Department of Economic Affairs of the Ministry of Finance is the GEF Political FP 19 The ToRs of the TWG refer to: planning, guiding and assessing technologies for final disposal of PCBs; and, making recommendations on best technologies to existing National Steering Committee on PCBs – source: TORs of the TC, MOEF Office Order 21 of 7 July 2010 20 Project Document (UNIDO India FSP PCBs-ProDoc-25 Nov09-resubfinal.pdf ) includes references to the Inception Workshop on 12 pages, and sets aside a budget of US 40,500 for this

37

this area are still to come with institutions further strengthened and/or self-sufficiency being acquired.

• To what extent have the expected outputs and outcomes been achieved or are likely to be achieved? How do the stakeholders perceive their quality? Were targeted beneficiary groups actually reached?

As detailed above, although to date the major outputs have not been completed, the ET considers that these will be achieved, contributing to, or setting the stage for the realization of the eventual outcomes of the project.

The stakeholders perceived quality as being in line with project expectations required to ensure the expected outputs and eventual outcomes. Interviewed stakeholders praised accomplishments and supported ongoing work.

With the exception of the private sector and civil society, the ET considers that target beneficiary groups were actually reached, in particular as regards the majority of the PCB-containing-transformer inventory holders.

• Are the actual project outcomes commensurate with the original or modified project objectives? If the original or modified expected results are merely outputs/inputs, the evaluators should assess if there were any real outcomes of the project and, if there were, determine whether these are commensurate with realistic expectations from such projects.

The results achieved at various levels so far, provide a well-laid foundation for taking off towards a promising and qualitative achievement of key project outcomes, in the full run phase. This will improve likeliness of achieving long-term objectives, which are expected to be commensurate with realistic expectations for results from this type of intervention.

• Identify the potential longer-term impacts or at least indicate the steps taken to assess these (see also below “monitoring of long term changes”). Wherever possible, evaluators should indicate how findings on impacts would be reported to the GEF in future.

Potential longer-term impacts of the Project are considered fully aligned with the expectations laid out in the original project document, as previously mentioned, and the steps taken to assess these are picked up in this MTE. Although this is yet to be demonstrated, the ET considers as likely the possibility that a PMS will be fully developed by Q1 2015 and that this will allow to track requirements and record progress. This will provide an indispensable and valuable source of data for the PSC, as well as for ulterior evaluations tasked with assessing the longer-term impacts of the Project.

• Catalytic or replication effects: the evaluation will describe any catalytic or replication effect of the project. If no effects are identified, the evaluation will describe the catalytic or replication actions that the project carried out. No ratings are requested for the project’s catalytic role.

Substantive replication possibilities are foreseen, which will likely be catalyzed by the issuance/enactment of regulatory guidelines/legislation. These regard mainly the technology, which the ET considers from the evidence received, moderately likely to likely to occur, via indigenization.

38

5.4 Efficiency Efficiency of the Project is assessed as Moderately Unsatisfactory. Although the outputs and activities that have been completed were implemented in a cost-effective and efficient manner, the planned/actual ratio of expenditures (70%) is low, at this stage in the life of the Project 21.

Although the low operational efficiency can be attributed to various types & levels of legal requirements, and the project is presently facing severe delays in its implementation and is highly unlikely to produce results within the time frame available (i.e. by Dec 2015), the ET considers that there are numerous mitigating factors. In particular, this low financial efficiency can not-only be linked to the fact that it is the first pilot of its kind (no prior country experience), but can be put in perspective by the fact that the likely project outcomes, in synergy with the GoI schemes, may lead to improvement of efficiency in the long term.

• Is the project cost effective? Is the project the least cost option? The ET was not informed of any concerns regarding cost effectiveness or costliness, and considers that although to date the project has made limited progress towards the delivery of results, this has been at a reasonable cost.

• Has the project produced results (outputs and outcomes) within the expected time frame? Was project implementation delayed, and, if it was, did that affect cost effectiveness or results? Wherever possible, the evaluator should also compare the costs incurred and the time taken to achieve outcomes with that for similar projects. Are the project’s activities in line with the schedule of activities as defined by the project team and annual work plans? Are the disbursements and project expenditures in line with budgets?

At this stage, the project has missed the planned deadlines and is yet to produce some of the major expected results. However, although this can be attributed in part to a series of “unfortunate events”; to a heavy dose of impossible-to-quantify “bad timing” of adverse events and decisions (see section on Effectiveness), and to country specific administrative constraints, the ET must conclude that the implementation of the project has been delayed.

The severity of the delays required the negotiation of an extension of the projects duration and even though the overall cost-effectiveness does not appear to be at stake yet, this is a risk that must be kept in check by the project team.

• Have the inputs from the donor, UNIDO and Government/counterpart been provided as planned, and were they adequate to meet requirements? Was the quality of UNIDO inputs and services as planned and timely?

The ET was not informed of any shortcomings or delays as regards provision of inputs from Government counterparts or from UNIDO and, contributions have reportedly been made in a timely manner.

21 The Project completion date is December of 2014, which as at time of this evaluation would leave only 2 months to expend 30% (US$ 4,243,180) of the planned budget

39

Table 1 - Overview of available funds

Cofinancing GEF Grant Total

Outcome 1: Strengthening of policy, legal, and regulatory framework 551,000 682,450 1,233,450

Outcome 2: Institutional capacity building and awareness raising 7,266,600 3,130,050 10,346,650

Outcome 3: Targeted regional implementation for ESM of PCBs, PCB-containing equipment and wastes

5,951,500 1,182,500 7,234,000

Outcome 4: Regional capability for final treatment and disposal of PCBs, PCB-containing equipment and wastes

15,053,400 8,643,000 23,696,400

Outcome 5: Project management 177,500 462,000 589,500

Total 29,000,000 14,100,000 43,100,000

Source: Terms of reference for the Evaluation (from Prodoc)

Table 2 – Expenditures of GEF grant as at November 2014

Total Allotment (US$)

Disbursement (US$)

Unliquidated Obligation

(US$)

Uncommitted Balance (US$)

International Experts 644,013.25 181,467.17 59,980.19 402,564.89

National Experts 925,248.75 314,936.20 28,902.53 581,950.02

Local travels 535,686.16 72,102.17 28,568.45 435,015.54

Sub-contracts 11,510,908.03 1,520,576.29 7,572,241.88 2,442,218.45

Training 226,256.66 7,000.37 226.39 219,029.90

International Meetings

91,265.68 45,648.37 1306.29 44,219.29

Equipment 6550.79 4,750.81 800 999.98

Miscellaneous 160,070.71 32,888.52 1375.52 117,182.19

Total 14,100,000.3 2,179,369.90 7,633,421.06 4,243,180.26

Source: Terms of reference for the Evaluation

Released budget Expenditures Funds Available

40

Table 3 – Expenditures of GEF grant as at

January 2015

Source: UNIDO

5.5 Sustainability The ET assesses the medium term Sustainability of the Project as Likely to Highly Likely. Based on the evidences provided, completion of the ongoing project activities is likely to take place within the time allotted by the project extension 22. Sustainability at that stage will be first and foremost linked to the approval of a regulatory framework, and then to the successful demonstration of the technologies. Both of these are considered highly likely to occur in the short to medium term.

• Financial, Socio-political, Institutional Framework and Governance, and Environmental Risks

The ET considers financial risks as Moderate to High as in particular, issues related to the urgent construction of the pilot plant at the SAIL facilities could, if derailed, see the withdrawal from the agreement of the selected technology provider.

The sustainability of the project will also require the strong participation of the private sector, as well as that of civil society agencies and institutions as their continued interest in the results will help to maintain the momentum, and pressure in favour of the long term continuity of the results. In this sense, the dissemination of the project’s results as a whole and of the pilot projects will be a catalyst to encourage the private sector and civil society to appropriate themselves of the project, contributing to sustainable results.

Overall the ET considers sociopolitical and Environmental risks to be Low.

5.6 Assessment of M&E systems • M&E design and implementation The ET was able to ascertain that although a carefully designed M&E system is included in the Project Document, implementation of this system has not started and hence monitoring of changes is not, at this stage possible. Overall the assessment of the M&E systems is therefore Highly Unsatisfactory.

Full deployment of an M&E system needs to be addressed with urgency as it is the opinion of the ET that although the opportunity to organize an Inception Workshop has long passed, implementation of all other elements detailed in the Section F of the

22 Strong justification for an extension has been submitted by the MOEF and the ET understands, based on this, that an 18 month extension has been granted

14,100,000 9,964,912 4,135,087

41

Project Document 23 will greatly benefit and contribute to the successful completion of the project.

This said, it is important to point out that the GoI is very clear in its objective of full deployment of such a system at the earliest possible time and the ET received strong assurances that this, and the newly understood availability of funds reserved for this purpose, would allow the system to be fully implemented, as staffed as required to ensure the successful implementation of activities and the long term monitoring of results.

• Budgeting and Funding The budgeting & funding of M&E activities is as per acceptable standards.

5.7 Monitoring of Long Term Changes • Did this project contribute to the establishment of a long-term monitoring

system? If it did not, should the project have included such a component? What were the accomplishments and shortcomings in establishment of this system? Is the system sustainable—that is, is it embedded in a proper institutional structure and does it have financing? Is the information generated by this system being used as originally intended?

See above. The M&E system is not yet in place.

5.8 Ratings Overview – Project Performance as per GEF Criteria 24

Criterion Evaluator’s Summary Comments ET Rating

Attainment of project objectives and results (overall rating) Sub criteria (below)

The indicators tend towards a higher rating, however at this stage major outputs are not yet

delivered MU

Effectiveness MU Relevance HS Efficiency MU Sustainability of Project outcomes (overall rating) Sub criteria (below)

Indicators tend towards HL as country’s commitment level is high L-HL

Financial L-HL Socio Political L-HL Institutional framework and governance L-HL Ecological L-HL Monitoring and Evaluation (overall rating) Sub criteria (below)

Weaknesses require urgent attention

M&E Design HU M&E Plan Implementation (use for adaptive management)

HU

Budgeting and Funding for M&E activities S

23 Development of project indicators, annual project financial audits, tripartite review meetings, etc. 24 Please refer to Annex xx for detailed information on the rating system used

42

Criterion Evaluator’s Summary Comments ET Rating

UNIDO specific ratings No shortcomings were evidenced by the evaluation

Quality at entry HS Implementation approach HS UNIDO Supervision and backstopping HS Overall Rating MU

5.9 Assessment of processes affecting attainment of project results • Preparation and readiness. Are the project’s objectives and components

clear, practicable and feasible within its time frame? Are counterpart resources (funding, staff, facilities), and adequate project management arrangements in place at project entry?

Although the project objectives are assessed as clear, practicable and feasible, the evidence shows that the time frame allotted for their completion was not adequate. In particular this refers to the establishment and approval of a legal framework but can also be more broadly applied to the complex and multilayered administrative processes in place at the time of project implementation. This has been discussed more at length in the section on project design, above.

The ET did not gather any evidence regarding untimely availability of counterpart resources at project entry, nor of inadequacy of project management arrangements.

• Were lessons from other relevant projects properly incorporated in the project design? Were the partnership arrangements properly identified and roles and responsibilities negotiated prior to project approval? Were counterpart resources (funding, staff, and facilities), enabling legislation, and adequate project management arrangements in place at project entry?

Lessons from other similar projects in Asia as well as Europe were properly incorporated in the project design and also during the project implementation phase. Partnership arrangements with the key project stakeholders were properly identified and roles and responsibilities negotiated prior to project approval. However during the initial years of project implementation the same were redesigned to adjust with CPRIs resource requirements. Also at the time of project entry, the counterpart resource, enabling legislation and adequate project management arrangements were reasonably well in place.

• Stakeholder involvement. Did the project involve the relevant stakeholders through information sharing and consultation and by seeking their participation in project design, implementation, and M&E? For example, did the project implement appropriate outreach and public awareness campaigns? Did the project consult with and make use of the skills, experience, and knowledge of the appropriate government entities, nongovernmental organizations, community groups, private sector entities, local governments, and academic institutions in the design, implementation, and evaluation of project activities? Were perspectives of those who would be affected by project decisions, those who could affect the outcomes, and those who could contribute information or other resources to the process taken into account while taking decisions? Were the relevant vulnerable groups and powerful supporters and opponents of the processes properly involved?

43

Project so far has involved the necessary and relevant stakeholders that are required to operationalize the project and put in place a structure and mechanism to achieve a successful and replicable pilot, which is a prime requisite for productively involving a wider set of relevant stakeholders.

• Financial planning. Did the project have the appropriate financial controls,

including reporting and planning, that allowed management to make informed decisions regarding the budget and allowed for timely flow of funds? Was there due diligence in the management of funds and financial audits? Did promised cofinancing materialize?

The project had the proper financial controls to allow management to make informed decisions regarding the budget as well as the flow of funds. As mentioned below under the M&E section, at the time of the MTE, no financial audits had been carried out.

The ET did not document any shortcomings regarding the cofinancing.

• UNIDO supervision and backstopping. Did UNIDO staff identify problems in a

timely fashion and accurately estimate their seriousness? Did UNIDO staff provide quality support and advice to the project, approve modifications in time, and restructure the project when needed? Did UNIDO provide the right staffing levels, continuity, skill mix, and frequency of field visits for the project?

UNIDO staff identified problems in a timely fashion and accurately estimated their seriousness. The ET was able to document the numerous interventions of UNIDO staff without which, in the words of a high-ranking official “this project would not have achieved even these limited results”.

• Cofinancing and project outcomes and sustainability. If there was a

difference in the level of expected cofinancing and the cofinancing actually realized, what were the reasons for the variance? Did the extent of materialization of cofinancing affect project outcomes and/or sustainability, and, if so, in what ways and through what causal linkages?

The co-financing commitments are either in place or well on the way to being in place, and overall there has been no modification of the project objectives.

UNIDO’s in-kind contribution to the project comprised the establishment of a project focal point and the provision of part-time assistance of senior staff to ensure the effective implementation of the project and to support project implementation, as well as the part-time assistance of the UNIDO RENPAP Regional Coordinator.

However, at the beginning targeted beneficiaries wanted to address state electricity boards, did not work and identified steel authority, project was reoriented on the new target beneficiaries, biggest inventory owner.

• Delays and project outcomes and sustainability. If there were delays in

project implementation and completion, what were the reasons? Did the delays affect project outcomes and/or sustainability, and, if so, in what ways and through what causal linkages?

Project is facing delays on multiple accounts, which in turn caused delays in project implementation within the allotted time duration of 5 years. Various reasons for delay are - engagement of new target beneficiary, changes in the leadership at the

44

partnership organization, ambiguity over the financial control, unforeseen failure in the procurement process (bid failure), and long time taken in the governmental approval processes. These are covered more extensively above.

5.10 Project Coordination and Management • Were the national management and overall coordination mechanisms

efficient and effective? Did each partner have specific roles and responsibilities from the beginning? Did each partner fulfill its role and responsibilities (e.g. providing strategic support, monitoring and reviewing performance, allocating funds, providing technical support, following up agreed/corrective actions…)?

The national management and overall co-ordination mechanisms were efficient and effective within the overall perspective of the prevailing national constraints and intra organizational operational complexities.

Changes in the leadership positions in some of the key partner organizations, viz. CPRI, MOEF, led to delays in the required implementation pace of the project. This coupled with the absence of a formal regular monitoring & reviewing mechanism further affected timely performance and fund flow of the project.

• Were the UNIDO HQ based management, coordination, quality control and technical inputs efficient, timely and effective (problems identified timely and accurately; quality support provided timely and effectively; right staffing levels, continuity, skill mix and frequency of field visits…)

Yes, UNIDO HQ and Field based management and coordination were provided in a timely and effective manner. Although the project outputs have not all been reached as expected, this was not due to lack of effort on their part and the ET was able to document the constant level of support and commitment provided by HQ and most importantly - as they are on the front line - of the Field office. The project also greatly benefitted from the extensive support and network of contacts of the RENPAP team (Regional Network for Pesticides Asia and the Pacific) that facilitated, on numerous occasions, the resolution of situations that could have derailed the project further, had they not been addressed rapid, diplomatic and efficiently.

5.11 Gender Mainstreaming • To which extent were socioeconomic benefits delivered by the project at

the national and local levels, including consideration of gender dimensions?

The main socioeconomic benefits will be delivered at the national level, but also at the local level in as much as components of the project are executed in specific localities. Overall the benefits will only directly affect a very small sector of the population, which is those involved with the electricity-generating sector. Although some level of employment might be generated, this is not considered a significant outcome.

The ET was not able to document gender related benefits specifically for women, however this can only be associated to the fact that at this stage, the project targets a sector that is predominantly male dominated (technicians, engineers in the electricity sector).

5.12 Procurement Issues No procurement related issues were brought to the attention of the ET during any of the interviews nor was the ET able to document any issues in the available documentation.

45

6 Conclusions and Recommendations

CONCLUSION 1

Maximizing Opportunities & Collaboration

Recommendation 1

Overall the Projects’ situation improved in terms of readiness but deteriorated in terms of time availability for completion

Project should take an urgent call on additional time requirements to achieve results


Project stakeholders are aware of delays/time lost and share key concerns on remaining activities and time available for completion

Once an extension is obtained, work plans & activities should follow a “war footing” approach

The project should reach out to other entities such as the Stockholm Convention Centre to facilitate timely coordination

CONCLUSION 2 Consolidating success

Recommendation 2

Lack of fully operational M&E systems within a weak PMU contributed to operational inefficiencies

PMU needs to be strengthened and regular monitoring and evaluation systems need to be formalized and fully implemented a.s.a.p.


The project suffered numerous unforeseen delays related but not limited to policy aspects i.e. approval of Notification, and administrative aspects i.e. Environmental Clearances, etc.

Reach-out /engagement of stakeholders was limited

There were changes of stakeholder representatives and even of stakeholders

A change of NPC occurred, however no proper handover took place, which hindered the efforts of the newly appointed NPC as he did not benefit from transfer of institutional memory

Effective coordination mechanisms need to be established to ensure strong commitment from each of the project stakeholders, in their independent objectives, to catch up on operational efficiency

Greater reach-out and engagement of stakeholders needs to be actively pursued, to ensure sustainability

Country should ensure proper hand over mechanisms are effectively utilized in times of transition including full briefings by the GEF FP

46

47

7 Lessons Learnt

Full implementation of a strong monitoring and review mechanism is primordial to ensure timely execution and completion of the targeted objectives.

Project design should incorporate flexibility/alternatives on the target beneficiary set, so that project does not face any setbacks on account of non-engagement of the target beneficiary within a given time frame.

Prior assessments of technology status in the host country should be carried out and accordingly the time and resources for the procurement process should be selected/proposed at the project design stage.

48

8 Annexes

1. GEF Rating of Project Objectives and Results 2. List of Interviewees 3. Bibliography 4. Terms of Reference for the Final Evaluation

49

8.1 GEF Ratings

RATINGS OF PROJECT OBJECTIVES AND RESULTS

• Highly Satisfactory (HS): The project had no shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

• Satisfactory (S): The project had minor shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

• Moderately Satisfactory (MS): The project had moderate shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

• Moderately Unsatisfactory (MU): The project had significant shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

• Unsatisfactory (U) The project had major shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

• Highly Unsatisfactory (HU): The project had severe shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

Please note: Relevance and effectiveness will be considered as critical criteria. The overall rating of the project for achievement of objectives and results may not be higher than the lowest rating on either of these two criteria. Thus, to have an overall satisfactory rating for outcomes a project must have at least satisfactory ratings on both relevance and effectiveness.

RATINGS ON SUSTAINABILITY

Sustainability will be understood as the probability of continued long-term outcomes and impacts after the GEF project funding ends. The evaluation will identify and assess the key conditions or factors that are likely to contribute or undermine the persistence of benefits beyond project completion. Some of these factors might be outcomes of the project, i.e. stronger institutional capacities, legal frameworks, socio-economic incentives /or public awareness. Other factors will include contextual circumstances or developments that are not outcomes of the project but that are relevant to the sustainability of outcomes.

Rating system for sustainability sub-criteria

On each of the dimensions of sustainability of the project outcomes will be rated as follows.

• Likely (L): There are no risks affecting this dimension of sustainability. • Moderately Likely (ML). There are moderate risks that affect this dimension of sustainability. • Moderately Unlikely (MU): There are significant risks that affect this dimension of

sustainability • Unlikely (U): There are severe risks that affect this dimension of sustainability.

All the risk dimensions of sustainability are critical. Therefore, overall rating for sustainability will not be higher than the rating of the dimension with lowest ratings. For example, if a project has an Unlikely rating in either of the dimensions then its overall rating cannot be higher than Unlikely, regardless of whether higher ratings in other dimensions of sustainability produce a higher average.

RATINGS OF PROJECT M&E

Monitoring is a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing project with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds. Evaluation is the systematic and objective assessment of an on-going or completed project, its design, implementation and results. Project evaluation may involve the definition of appropriate standards, the examination of performance against those standards, and an assessment of actual and expected results.

The Project monitoring and evaluation system will be rated on ‘M&E Design’, ‘M&E Plan Implementation’ and ‘Budgeting and Funding for M&E activities’ as follows:

50

• Highly Satisfactory (HS): There were no shortcomings in the project M&E system. • Satisfactory(S): There were minor shortcomings in the project M&E system. • Moderately Satisfactory (MS): There were moderate shortcomings in the project M&E

system. • Moderately Unsatisfactory (MU): There were significant shortcomings in the project M&E

system. • Unsatisfactory (U): There were major shortcomings in the project M&E system. • Highly Unsatisfactory (HU): The Project had no M&E system.

“M&E plan implementation” will be considered a critical parameter for the overall assessment of the M&E system. The overall rating for the M&E systems will not be higher than the rating on “M&E plan implementation.”

All other ratings will be on the GEF six-point scale.

HS = Highly Satisfactory Excellent S = Satisfactory Well above average MS = Moderately Satisfactory Average MU = Moderately Unsatisfactory Below Average U = Unsatisfactory Poor HU = Highly Unsatisfactory Very poor (Appalling)

51

8.2 List of Interviewees

Name Institution Position / Contact Dr SP Dhua UNIDO Regional Co-Ordinator - RENPAP

Email: [email protected] Phone: +91-11-24629112

Dr YP Ramdev UNIDO National Technical Advisor - RENPAP Email: [email protected] Phone: +91-11-42836114

Dr RN Jindal MoEF&CC Director Email: [email protected] Phone: +91-1124695325

Mr. W Bharat Singh MoEF&CC Joint Director Email: [email protected] Phone: +91-11-24695338

Mr. VV Pattanshetti CPRI Joint Director Email: [email protected] Phone: +91-80-23604448

Dr. Parvati Ramaswamy CPRI Senior Consultant (PCB Project) Email: [email protected] Phone: +91-80-23602919

Mr. Arun Bhandakkar SAIL - BSP General Manager (Electric) Email: [email protected] Phone: +91-788-2221386

Mr. RG Gupta SAIL-BSP Dy.. General Manager (Power Systems Dept.) Email: [email protected] Phone: +91-788-2854104

Mr. RK Agrawal SAIL-BSP Dy. General Manager (Environment Management) Email:[email protected] Phone: +91-788-2854728

Mr. Sunil Singal SAIL Asst. General Manager (EMD) Email: [email protected] Phone: +91-11-22443639

Dr. (Ms.) Meenakshi Kakkar SAIL General Manager (EMD) Email: [email protected]

Dr. K Srinivas Ramky Enviro Engineers Ltd.

Head – Planning & Technical Email: [email protected] Phone: +91-40-23015000

Mr. P Eshwar Reddy Ramky Enviro Engineers Ltd.

Director (Engineering) Email: [email protected] Phone: +91-40-44422222

Ms. Ayumi Fujino UNIDO UNIDO Representative and Regional Director for South Asia, Email: [email protected] Phone:+91-11-24643484

Ms. Carmela Centeno UNIDO Industrial Development Officer, Environmental Management Branch Email : [email protected]

Mr. Vinay Vij UNIDO Adminstrative Assistant Email: [email protected] Phone: +91-11-46532321

52

mailto:[email protected]













8.3 Bibliography

53

8.4 Terms of Reference for the MTE

UNITED NATIONS INDUSTRIAL DEVELOPMENT ORGANIZATION

Terms of Reference

Independent Mid-Term Evaluation of the UNIDO Project

Project Number: GFIND10001 SAP ID: 104044

Environmentally Sound Management and Final Disposal of PCBs in India

September 2014

54

Contents:

I. Project Background and overview 56 II. Scope and Purpose of the Evaluation 59 III. Evaluation Approach and Methodology 60 IV. Evaluation Team Composition 61 V. Time Schedule and Deliverables 62 VI. Project Evaluation Parameters 62 VII. Reporting 68 VIII. Quality Assurance 69

Annex 1 - Outline of an In-Depth Project Evaluation Report 70 Annex 2 - Overall Ratings Table 72 Annex 3 - GEF Minimum Requirements for M&E 75 Annex 4 – Required Project Identification and Financial Data 76 Annex 5. Job Descriptions 80 Annex 6 – UNIDO Procurement Process 85

55

9 I. Project Background and overview

Project origin and objectives The Republic of India signed the Stockholm Convention on POPs on 14 May 2002 and ratified it on 13 January 2006. The Global Environment Facility (GEF)-funded project entitled “Development of a National Implementation Plan (NIP) in India as a First Step to Implement the Stockholm Convention on Persistent Organic Pollutants (POPs)” was approved on 14 June 2007. By the Government of India’s notification, the Ministry of Environment, Forest and Climate Change (MoEF&CC) was assigned as the National Focal Point (NFP) for the Stockholm Convention on POPs. The MOEF has selected the environmentally sound Polychlorinated biphenyls (PCBs) management and disposal as one of the first priorities of post-NIP program. The reason to give priority to the PCBs sector was that its implementation timeframe was clearly defined by the Stockholm Convention. PCBs were never produced in India and their import was banned in 1998. The survey carried out by Central Power Research Institute (CPRI) from 2004 to 2008, which included establishment of a preliminary inventory on PCB-containing electrical equipment and the evaluation of India’s capacity for the Environmental Sound Management (ESM) and disposal of PCBs, partially assessed the PCB situation in India. In all 28 states and 7 union territories of India, an estimated total number of about 45,000 power and about 3,500,000 distribution transformers exist. While the total amount of imported PCB-oils as well as PCB-containing equipment is unknown, the estimated quantity of pure PCB-oils and PCB-containing equipment in India according to the initial inventory amounts to 7,700 tons. The project “Environmentally sound management and final disposal of PCBs in India” addresses national priorities such as improvement of the legislation on POPs chemicals, elimination of PCB-containing equipment, reduction of PCBs releases from industrial wastes and sewages, improvement of environmental performance in power sector, improvement of environmental performance in industry sector, and identification of PCBs wastes and contaminated sites and their environmentally sound and safe management.

It is a unique feature of this project that it was implemented simultaneously with the development and formulation of the relevant parts of the NIP, also as a GEF-funded Full-Sized Project (FSP). The national executing agency of this project is the Central Power Research Institute (CPRI) and CPRI is the government entity that developed and formulated the PCB-related chapters in the National Implementation Plan (NIP).

This project will focus on the reduction and elimination of PCBs in the power sector through: (a) development of appropriate legislation, (b) provision of capacity building for key stakeholders, (c) development of an Environmental Sound Management (ESM) system for PCBs, PCB-

containing electrical equipment and waste and incorporating it into a national policy framework,

(d) gradual phase-out of PCB-containing equipment (transformers and capacitors), (e) disposal of all PCBs, PCB-containing equipment and wastes, (f) strengthening environmental monitoring capacities, and

56

(g) identifying the most appropriate mitigation measures to reduce social costs of complying with the Stockholm Convention.

In addition, the project will start to create a national inventory of non-electrical equipment and other articles containing more than 0.005 percent of PCBs as required by the Stockholm Convention.

According to the Project Document, the expected outcomes were defined as follows:

Outcome 1: Strengthened policy and regulatory framework to comply with the obligations under the Stockholm Convention,

Outcome 2: Relevant institutions in India are enabled to manage PCBs in an environmentally sound manner as well as awareness raising on the adverse effects of PCBs,

Outcome 3: Targeted and regional implementation for ESM of PCBs, PCB containing equipment and wastes, and

Outcome 4: Regional Capacity for final treatment and disposal of PCB wastes.

Relevance to GEF programmes Pursuant to POPs focal area Strategy and Strategic Programming 1 and 2 for GEF-4 (2007-2010) approved by the GEF Council in September 2007, the GEF will finance projects aiming at strengthening capacities for NIP implementation in order to meet the country’s obligations under the Stockholm Convention including POPs reduction measures, and partnering in investments in NIP implementation that sustainably reduce POPs production, use and releases through phase-out, destruction in an environmentally sound manner, and use of substitute products and alternative practices that lead to reduced environmental and health risks caused by POPs. The project will support this work by application of ESM in the management and disposal of PCB-containing equipment, oil and wastes. Implementation arrangements UNIDO acted as the GEF Implementing Agency (IA) for the project. A project focal point is appointed at UNIDO to assist project execution. This focal point is a dedicated core staff, supplemented by support from support staff colleagues on a part-time as required basis, supervised by a senior professional staff engaged in the management and coordination of UNIDO’s Stockholm Convention Program. UNIDO will provide these services available as part of its in-kind contribution to the project.

The Ministry of Environment and Forests (MOEF) is the national executing partner for the project. It is the nodal agency for planning, promoting and coordinating environmental programmes including the management of chemical disasters in India. The Ministry is mandated to protect the land, air and water systems and is responsible for the prevention and control of pollution including hazardous substances. The MOEF is the GEF and Stockholm Convention focal point in the country, which coordinates activities and cooperation between relevant stakeholders of the NIP. MOEF is empowered to promulgate rules under the Environment Protection Act and is responsible for ensuring effective implementation of legislation, monitoring and control of pollution (including pesticide levels in soil and water), environmental clearances for industrial development projects, promotion of environmental education, training and awareness, and coordination with concerned agencies at the national and international level. MOEF establishes standard for the quality of the environment, including emissions and/or discharges of environmental pollutants from various sources. The ministry establishes procedures and safeguards for the prevention of accidents that may cause

57

environmental pollution. MOEF can issue direction for the closure and prohibition or regulation of an industry, operations or processes.

MOEF entrusted the Central Power Research Institute (CPRI) to be the national executing agency of this project.

The Joint Secretary of the Ministry has been assigned as National Project Director (NPD) and the Director has taken the position of the National Project Coordinator (NPC). In addition, two assistant project coordinators assist the POPs Management Unit.

The Technical Working Group consists of a representative of MOEF, a representative of Central Electricity Authority (CEA) on behalf of the Ministry of Power (MOP), a representative of Central Pollution Control Board (CPCB), representatives of major stakeholder utilities (state electricity boards), representative of CPRI as the national executing agency, the National Technical Advisor (NTA), the Chief Technical Advisor (CTA), and the UNIDO project manager.

The project recruited an international Chief Technical Advisor (CTA), a National Technical Advisor (NTA), policy experts, PCBs management and disposal industry experts, chemists, monitoring & evaluation experts and other technical experts as required.

Private industries and entrepreneurs can be roped in to establish the common facilities for disposal of PCBs. Private sector stakeholders and other potential project participants will be actively recruited and integrated into the project.

Implementation Status The project was approved by the GEF in January 2010 but the inception workshop was held only in August 2010.

Main project stakeholders The MOEF is the government coordinating agency for the project. In addition, two key stakeholder agencies play crucial roles in the project. The first is the Ministry of Power (MOP), the counterpart agency, which is responsible for assisting in the implementation of the activities and measures for limitation, elimination and monitoring of import and use of PCB-containing equipment and reduction of unintentional production of POPs chemicals. The other key stakeholder is the CPRI, which is the national executing agency/cooperating agency for the project. CPRI organizes and carries out all capacity building activities related to technicalities of the environmentally sound PCBs management of the PCB-containing electrical equipment in the country. BUDGET INFORMATION a) Overall Cost and Financing (including co-financing):

Cofinancing GEF Grant Total

Outcome 1: Strengthening of policy, legal, and regulatory framework 551,000 682,450 1,233,450

Outcome 2: Institutional capacity building and awareness raising 7,266,600 3,130,050 10,346,650

Outcome 3: Targeted regional implementation for ESM of PCBs, PCB-containing equipment and wastes

5,951,500 1,182,500 7,234,000

58

Outcome 4: Regional capability for final treatment and disposal of PCBs, PCB-containing equipment and wastes

15,053,400 8,643,000 23,696,400

Outcome 5: Project management 177,500 462,000 589,500

Total 29,000,000 14,100,000 43,100,000

Source: Project document

b) UNIDO budget (GEF funding excluding agency support cost):

Total Allotment (US$)

Disbursement (US$)

Unliquidated Obligation

(US$)

Uncommitted Balance (US$)

International Experts 644,013.25 181,467.17 59,980.19 402,564.89

National Experts 925,248.75 314,936.20 28,902.53 581,950.02

Local travels 535,686.16 72,102.17 28,568.45 435,015.54

Sub-contracts 11,510,908.03 1,520,576.29 7,572,241.88 2,442,218.45

Training 226,256.66 7,000.37 226.39 219,029.90

International Meetings

91,265.68 45,648.37 1306.29 44,219.29

Equipment 6550.79 4,750.81 800 999.98

Miscellaneous 160,070.71 32,888.52 1375.52 117,182.19

Total 14,100,000.3 2,179,369.90 7,633,421.06 4,243,180.26

10 II. Scope and Purpose of the Evaluation The mid-term evaluation will cover the duration of the project from its starting date in January 2010 to the estimated mid-term evaluation date September 2014. It will assess project performance and progress against the evaluation criteria: relevance, effectiveness, efficiency, sustainability and impact. The purpose of the mid-term evaluation is to enable the Government, counterparts, the GEF, UNIDO and other stakeholders and donors to:

(a) Verify prospects for development impact and sustainability, providing an analysis of the attainment of global environmental objectives, project objectives, delivery and completion of project outputs/activities, and outcomes/impacts based on indicators. The assessment includes re-examination of the relevance of the objectives and other

59

elements of project design according to the project evaluation parameters defined in chapter V.

(b) Enhance project relevance, effectiveness, efficiency and sustainability by proposing a set of recommendations with a view to ongoing and future activities.

(c) Draw lessons of wider applicability for the replication of the experience gained in this project in other projects/countries.

The key question of the evaluation is whether the project have made a significant contribution to reducing the effects of POPs on human health and the environment, i.e. whether the project is going to reduce and eliminate the use and releases of PCBs to the environment through promotion of measures to minimize exposures and risks by introducing environmentally sound management and disposal of PCBs, PCB-containing equipment and PCB-containing mineral oils and wastes aiming at the final and virtual disposal of all PCBs inventory in India by 2025 and 2028, respectively.

11 III. Evaluation Approach and Methodology The mid-term evaluation will be conducted in accordance with the UNIDO Evaluation Policy, the UNIDO Guidelines for the Technical Cooperation Programmes and Projects, the GEF’s 2008 Guidelines for Implementing and Executing Agencies to Conduct Terminal Evaluations, the GEF Monitoring and Evaluation Policy from 2010 and the Recommended Minimum Fiduciary Standards for GEF Implementing and Executing Agencies. It will be carried out as an independent in-depth evaluation using a participatory approach whereby all key parties associated with the project are kept informed and regularly consulted throughout the evaluation. The evaluation team leader will liaise with the Project Manager on the conduct of the evaluation and methodological issues. The evaluation team will be required to use different methods to ensure that data gathering and analysis deliver evidence-based qualitative and quantitative information, based on diverse sources: desk studies and literature review, statistical analysis, individual interviews, focus group meetings, surveys and direct observation. This approach will not only enable the evaluation to assess causality through quantitative means but also to provide reasons for why certain results were achieved or not and to triangulate information for higher reliability of findings. The concrete mixed methodological approach will be described in the inception report. The evaluation team will develop interview guidelines. Field interviews can take place either in the form of focus-group discussions or one-to-one consultations. The methodology will be based on the following: 1. A desk review of project documents including, but not limited to:

(a) The original project document, monitoring reports (such as progress and financial reports to UNIDO and GEF annual Project Implementation Review reports), output reports (case studies, action plans, sub-regional strategies, etc.) and relevant correspondence.

(b) Reports from the Project Expert Team (PET), Steering Group meetings and consultants.

(c) Other project-related material produced by the project.

2. The evaluation team will use available models of (or reconstruct if necessary) theory of change for the different types of intervention (enabling, capacity, investment, demonstration). The validity of the theory of change will be examined through specific questions in interviews and possibly through a survey of stakeholders.

60

3. Counterfactual information: In those cases where baseline information for relevant

indicators is not available the evaluation team will aim at establishing a proxy-baseline through recall and secondary information.

4. Interviews with project management and technical support including Ms. Carmela

Centeno, UNIDO Project Manager and staff associated with the project’s administrative and financial administration, if necessary.

5. Interviews with project managers, national experts, partner facilities and universities. 6. On-site observation of results achieved in demonstration projects, including interviews

of actual and potential beneficiaries of improved technologies and practices. 7. Interviews and telephone interviews with intended users for the project outputs and

other stakeholders involved with this project. The evaluator shall determine whether to seek additional information and opinions from representatives of any donor agencies or other organisations.

8. Interviews with the UNIDO Country Offices staff involved in the project as necessary. 9. Other interviews, surveys or document reviews as deemed necessary by the evaluator

and/or UNIDO EVA.

10. The inception report will provide details on the methodology used by the evaluation

team and include an evaluation matrix.

IV. Evaluation Team Composition

The evaluation team will be composed of one international evaluation consultant acting as a team leader and one national evaluation consultant.

The evaluation team should be able to provide information relevant for follow-up studies, including evaluation verification on request to the GEF partnership up to two years after completion of the evaluation.

The evaluation team will be contracted by UNIDO. The tasks of each team member are specified in the job descriptions attached to these terms of reference.

Members of the evaluation team must not have been directly involved in the design and/or implementation of the programme/projects.

The Project Manager at UNIDO and the Government of India will support the evaluation team. The UNIDO GEF Coordinator will be briefed on the evaluation and equally provide support to its conduct. The UNIDO GEF Coordinator will be briefed on the evaluation.

61

V. Time Schedule and Deliverables

The mid-term evaluation is scheduled to take place in the period from September 2014 to December 2014. The field mission is planned for September 2014. At the end of the field mission, there will be a presentation of the preliminary findings for all stakeholders involved in this project in India.

After the field mission, the evaluation team leader will present the preliminary findings at UNIDO HQ. The draft mid-term evaluation report will be submitted 4-6 weeks after the end of the field mission.

VI. Project Evaluation Parameters

The ratings for the parameters described in the following sub-chapters A to I will be presented in the form of a table with each of the categories rated separately and with brief justifications for the rating based on the findings of the main analysis. An overall rating for the project should also be given. The rating system to be applied is specified in Annex 2.

A. Project design The evaluation will examine the extent to which:

the project’s design is adequate to address the problems at hand; a participatory project identification process was instrumental in selecting problem

areas and national counterparts; the project has a clear thematically focused development objective, the attainment of

which can be determined by a set of verifiable indicators; the project was formulated based on the logical framework (project results

framework) approach; the project was formulated with the participation of national counterpart and/or target

beneficiaries; and relevant country representatives (from government, industries and civil society) have

been appropriately involved and were participating in the identification of critical problem areas and the development of technical cooperation strategies.

B. Project relevance

The evaluation will examine the extent to which the project is relevant to the: national development and environmental priorities and strategies of the Government

and population of India, and regional and international agreements. See possible evaluation questions under “Country ownership/driven-ness” below.

target groups: relevance of the project’s objectives, outcomes and outputs to the different target groups of the interventions (e.g. companies, civil society, beneficiaries of capacity building and training, etc.).

GEF’s focal areas/operational programme strategies: In retrospect, were the project’s outcomes consistent with the focal areas/operational program strategies

62

of GEF? Ascertain the likely nature and significance of the contribution of the project outcomes to the wider portfolio of GEF’s Operational Program #14 concerning Persistent Organic Pollutants (POPs) .

UNIDO’s thematic priorities: Were they in line with UNIDO’s mandate, objectives and outcomes defined in the Programme & Budget and core competencies?

Does the project remain relevant taking into account the changing environment? Is there a need to reformulate the project design and the project results framework given changes in the country and operational context?

C. Effectiveness: attainment of objectives and planned results (progress to

date):

Assessment of project outcomes should be a priority:

• What outputs and outcomes has the project achieved so far (both qualitative and quantitative results)? Has the project generated any results that could lead to changes of the assisted institutions? Have there been any unplanned effects?

• The evaluation will assess to what extent results at various levels, including outcomes, have been achieved. In detail, the following issues will be assessed: To what extent have the expected outputs, outcomes and long-term objectives been achieved or are likely to be achieved?

• Are the actual project outcomes commensurate with the original or modified project objectives? If the original or modified expected results are merely outputs/inputs, the evaluators should assess if there were any real outcomes of the project and, if there were, determine whether these are commensurate with realistic expectations from such projects.

• To what extent have the expected outputs and outcomes been achieved or are likely to be achieved? How do the stakeholders perceive their quality? Were the targeted beneficiary groups actually reached?

• Identify the potential longer-term impacts or at least indicate the steps taken to assess these (see also below “monitoring of long term changes”). Wherever possible, evaluators should indicate how findings on impacts will be reported to the GEF in future.

.

• Catalytic or replication effects: the evaluation will describe any catalytic or replication effect of the project. If no effects are identified, the evaluation will describe the catalytic or replication actions that the project carried out. No ratings are requested for the project’s catalytic role.

D. Efficiency The extent to which:

• The project cost was effective? Was the project using the least cost options?

63

• Has the project produced results (outputs and outcomes) within the expected time frame? Was project implementation delayed, and, if it was, did that affect cost effectiveness or results? Wherever possible, the evaluator should also compare the costs incurred and the time taken to achieve outcomes with that for similar projects. Are the project’s activities in line with the schedule of activities as defined by the project team and annual work plans? Are the disbursements and project expenditures in line with budgets?

• Have the inputs from the donor, UNIDO and Government/counterpart been provided as planned, and were they adequate to meet requirements? Was the quality of UNIDO inputs and services as planned and timely? • Was there coordination with other UNIDO and other donors’ projects, and did possible synergy effects happen?

E. Assessment of sustainability of project outcomes:

Sustainability is understood as the likelihood of continued benefits after the GEF project ends. Given the uncertainties involved, it may be difficult to have a realistic a priori assessment of sustainability of outcomes. Therefore, assessment of sustainability of outcomes will give special attention to analysis of the risks that are likely to affect the persistence of project outcomes. This assessment should explain how the risks to project outcomes will affect continuation of benefits after the GEF project ends. It will include both exogenous and endogenous risks. The following four dimensions or aspects of risks to sustainability will be addressed:

a. Financial risks. Are there any financial risks that may jeopardize sustainability of project outcomes? What is the likelihood of financial and economic resources not being available once GEF assistance ends? (Such resources can be from multiple sources, such as the public and private sectors or income-generating activities; these can also include trends that indicate the likelihood that, in future, there will be adequate financial resources for sustaining project outcomes.)

b. Sociopolitical risks. Are there any social or political risks that may jeopardize sustainability of project outcomes? What is the risk that the level of stakeholder ownership (including ownership by governments and other key stakeholders) will be insufficient to allow for the project outcomes/benefits to be sustained? Do the various key stakeholders see that it is in their interest that project benefits continue to flow? Is there sufficient public/stakeholder awareness in support of the project’s long-term objectives?

c. Institutional framework and governance risks. Do the legal frameworks, policies, and governance structures and processes within which the project operates pose risks that may jeopardize sustainability of project benefits? Are requisite systems for accountability and transparency, and required technical know-how, in place?

d. Environmental risks. Are there any environmental risks that may jeopardize sustainability of project outcomes? The evaluation should assess whether certain activities will pose a threat to the sustainability of the project outcomes. For example, construction of a dam in a protected area could inundate a sizable area and thereby neutralize the biodiversity-related gains made by the project.

64

F. Assessment of monitoring and evaluation systems and project management: • M&E design. Does the project have a sound M&E plan to monitor results and track

progress towards achieving project objectives? The Evaluation will assess whether the project met the minimum requirements for the application of the Project M&E plan (see Annex 3).

• M&E implementation. The evaluation should verify that an M&E system was in place and facilitated timely tracking of progress toward project objectives by collecting information on chosen indicators continually throughout the project implementation period; annual project reports were complete and accurate, with well-justified ratings; the information provided by the M&E system was used during the project to improve performance and to adapt to changing needs; and projects had an M&E system in place with proper training for parties responsible for M&E activities to ensure that data will continue to be collected and used after project closure.

• Budgeting and Funding for M&E activities. In addition to incorporating information on funding for M&E while assessing M&E design, the evaluators will determine whether M&E was sufficiently budgeted for at the project planning stage and whether M&E was funded adequately and in a timely manner during implementation.

• Monitoring of Long-Term Changes. The monitoring and evaluation of long-

term changes is often incorporated in GEF-supported projects as a separate component and may include determination of environmental baselines; specification of indicators; and provisioning of equipment and capacity building for data gathering, analysis, and use. This section of the evaluation report will describe project actions and accomplishments toward establishing a long-term monitoring system. The review will address the following questions: a. Did this project contribute to the establishment of a long-term monitoring

system? If it did not, should the project have included such a component? b. What were the accomplishments and shortcomings in establishment of this

system? c. Is the system sustainable—that is, is it embedded in a proper institutional

structure and does it have financing? d. Is the information generated by this system being used as originally

intended? • Project management. Were the national management and overall coordination

mechanisms efficient and effective? Did each partner have specific roles and responsibilities from the beginning? Did each partner fulfill its role and responsibilities (e.g. providing strategic support, monitoring and reviewing performance, allocating funds, providing technical support, following up agreed/corrective actions…)? Were the UNIDO HQ based management, coordination, quality control and technical inputs efficient, timely and effective (problems identified timely and accurately; quality support provided timely and effectively; right staffing levels, continuity, skill mix and frequency of field visits…)

• Implementation approach25. Is the implementation approach chosen different from other implementation approaches applied by UNIDO and other agencies? Does the approach comply with the principles of the Paris Declaration? Does the approach promote local ownership and capacity building? Does the approach involve significant risks?

25 Implementation approach refers to the concrete manifestation of cooperation between UNIDO, Government counterparts and local implementing partners. Usually POPs projects apply a combination of agency execution (direct provision of services by UNIDO) with elements of national execution through sub-contracts.

65

G. Assessment of processes affecting attainment of project results The evaluation will consider, but need not be limited to, the following issues that may have affected project implementation and attainment of project results:

a. Preparation and readiness. Were the project’s objectives and components clear, practicable, and feasible within its time frame? Were the capacities of the executing institution(s) and its counterparts properly considered when the project was designed? Were lessons from other relevant projects properly incorporated in the project design? Were the partnership arrangements properly identified and roles and responsibilities negotiated prior to project approval? Were counterpart resources (funding, staff, and facilities), enabling legislation, and adequate project management arrangements in place at project entry?

b. Country ownership/drivenness. Was the project concept in line with the sectoral and development priorities and plans of the country—or of participating countries, in the case of multicountry projects? Are project outcomes contributing to national development priorities and plans? Were the relevant country representatives from government and civil society involved in the project? Did the recipient government maintain its financial commitment to the project? Has the government—or governments in the case of multicountry projects—approved policies or regulatory frameworks in line with the project’s objectives?

c. Stakeholder involvement. Did the project involve the relevant stakeholders through information sharing and consultation and by seeking their participation in project design, implementation, and M&E? For example, did the project implement appropriate outreach and public awareness campaigns? Did the project consult with and make use of the skills, experience, and knowledge of the appropriate government entities, nongovernmental organizations, community groups, private sector entities, local governments, and academic institutions in the design, implementation, and evaluation of project activities? Were perspectives of those who would be affected by project decisions, those who could affect the outcomes, and those who could contribute information or other resources to the process taken into account while taking decisions? Were the relevant vulnerable groups and powerful supporters and opponents of the processes properly involved?

d. Financial planning. Did the project have the appropriate financial controls, including reporting and planning, that allowed management to make informed decisions regarding the budget and allowed for timely flow of funds? Was there due diligence in the management of funds and financial audits? Did promised cofinancing materialize?

e. UNIDO supervision and backstopping. Did UNIDO staff identify problems in a timely fashion and accurately estimate their seriousness? Did UNIDO staff provide quality support and advice to the project, approve modifications in time, and restructure the project when needed? Did UNIDO provide the right staffing levels, continuity, skill mix, and frequency of field visits for the project?

f. Cofinancing and project outcomes and sustainability. If there was a difference in the level of expected cofinancing and the cofinancing actually realized, what were the reasons for the variance? Did the extent of materialization of cofinancing affect project outcomes and/or sustainability, and, if so, in what ways and through what causal linkages?

g. Delays and project outcomes and sustainability. If there were delays in project implementation and completion, what were the reasons? Did the delays

66

affect project outcomes and/or sustainability, and, if so, in what ways and through what causal linkages?

The evaluation team will rate the project performance as required by the GEF. The ratings will be given to four criteria: Project Results, Sustainability, Monitoring and Evaluation, and UNIDO related issues as specified in Annex 2. The ratings will be presented in a table with each of the categories rated separately and with brief justifications for the rating based on the findings of the main analysis. An overall rating for the project should also be given. The rating system to be applied is specified in the same annex. As per the GEF’s requirements, the report should also provide information on project identification, time frame, actual expenditures, and co-financing in the format in Annex 4, which is modeled after the GEF’s project identification form (PIF).

H. Assessment of processes affecting attainment of project results The evaluation will consider, but need not be limited to, the following issues that may have affected gender mainstreaming in the project:

a. To which extent were socioeconomic benefits delivered by the project at the national and local levels, including consideration of gender dimensions?

I. Procurement issues The following evaluation questions that will feed in the Thematic Evaluation on Procurement have been developed and would be included as applicable in all projects (for reference, please see Annex 6 of the ToR: UNIDO Procurement Process):

• To what extent does the process provide adequate treatment to different types of procurement (e.g. by value, by category, by exception…)

• Was the procurement timely? How long the procurement process takes (e.g. by value, by category, by exception…)

• Did the good/item(s) arrive as planned or scheduled? If no, how long were the times gained or delays. If delay, what was the reason(s)?

• Were the procured good(s) acquired at a reasonable price? • To what extent were the procured goods of the expected/needed quality and

quantity? • Were the transportation costs reasonable and within budget. If no, pleased

elaborate. • Was the freight forwarding timely and within budget?. If no, pleased elaborate. • Who was responsible for the customs clearance? UNIDO FO? UNDP?

Government? Other? • Was the customs clearance handled professionally and in a timely manner?

How many days did it take? • How long time did it take to get approval from the government on import duty

exemption? • Which were the main bottlenecks / issues in the procurement process? Which

good practices have been identified? To what extent roles and responsibilities of the different stakeholders in the different procurement stages are established, adequate and clear?

• To what extent there is an adequate segregation of duties across the procurement process and between the different roles and stakeholders?

67

VII. Reporting

Inception report This Terms of Reference provides some information on the evaluation methodology but this should not be regarded as exhaustive. After reviewing the project documentation and initial interviews with the project manager the International Evaluation Consultant will prepare, in collaboration with the national consultant, a short inception report that will operationalize the ToR relating to the evaluation questions and provide information on what type of and how the evidence will be collected (methodology). The Inception Report will focus on the following elements: preliminary project theory model(s); elaboration of evaluation methodology including quantitative and qualitative approaches through an evaluation framework (“evaluation matrix”); division of work between the International Evaluation Consultant and National Consultant; mission plan, including places to be visited, people to be interviewed and possible surveys to be conducted and a debriefing and reporting timetable26.

Evaluation report format and review procedures

The draft report will be delivered to UNIDO EVA (the suggested report outline is in Annex 1) and circulated to UNIDO staff and national stakeholders associated with the project for factual validation and comments. Any comments or responses, or feedback on any errors of fact to the draft report provided by the stakeholders will be sent to the Project Manager for collation and onward transmission to the project evaluation team who will be advised of any necessary revisions. On the basis of this feedback, and taking into consideration the comments received, the evaluation team will prepare the final version of the mid-term evaluation report.

The evaluation team will present its preliminary findings to the local stakeholders at the end of the field visit and take into account their feed-back in preparing the evaluation report. A presentation of preliminary findings will take place in September/October 2014 and at HQ after the field mission.

The mid-term evaluation report should be brief, to the point and easy to understand. It must explain the purpose of the evaluation, exactly what was evaluated, and the methods used. The report must highlight any methodological limitations, identify key concerns and present evidence-based findings, consequent conclusions, recommendations and lessons. The report should provide information on when the evaluation took place, the places visited, who was involved and be presented in a way

26 The evaluator will be provided with a Guide on how to prepare an evaluation inception report prepared by the UNIDO Office for Independent Evaluation.

68

that makes the information accessible and comprehensible. The report should include an executive summary that encapsulates the essence of the information contained in the report to facilitate dissemination and distillation of lessons.

Findings, conclusions and recommendations should be presented in a complete, logical and balanced manner. The evaluation report shall be written in English and follow the outline given in Annex 1.

Evaluation Work Plan

The “Evaluation Work Plan” includes the following main products:

1. Desk review, briefing by project manager and development of methodology: Following the receipt of all relevant documents, and consultation with the Project Manager about the documentation, including reaching an agreement on the Methodology, the desk review could be completed.

2. Inception report: At the time for departure to the field mission, the complete gamete of received materials have been reviewed and consolidated into the Inception report.

3. Field mission: The principal responsibility for managing this evaluation lies with UNIDO. It will be responsible for liaising with the project team to set up the stakeholder interviews, arrange the field missions, coordinate with the Government. At the end of the field mission, there will be a presentation of preliminary findings to the key stakeholders in the country where the project was implemented.

4. Preliminary findings from the field mission: Following the field mission, the

main findings, conclusions and recommendations would be prepared and presented in the field and at UNIDO Headquarters.

5. A draft Mid-term evaluation report will be forwarded electronically to the Project Manager, who will forward the same to the Office for Independent Evaluation and circulated to main stakeholders.

6. A final Mid-term evaluation report will incorporate comments received.

VIII. Quality Assurance

The Project Manager (PM) will be responsible for managing the evaluation, preparing the terms of reference (TOR) and the job description (JD) of the evaluation consultant(s) on the basis of guidance of UNIDO’s Office for Independent Evaluation (ODG/EVA). The PM will forward drafts and final reports to ODG/EVA for review, distribute drafts and final reports to stakeholders (upon review by ODG/EVA), and organize presentations of preliminary evaluation findings which serve to generate feedback on and discussion of evaluation findings and recommendations at UNIDO HQ. Finally, the PM will be responsible for the submission of the final Mid-Term Evaluation Report to the GEF and to ODG/EVA.

69

11.1 Annex 1 - Outline of an In-Depth Project Evaluation Report Executive summary

Must provide a synopsis of the storyline which includes the main evaluation findings and recommendations

Must present strengths and weaknesses of the project Must be self-explanatory and should be 3-4 pages in length

I. Evaluation objectives, methodology and process Information on the evaluation: why, when, by whom, etc. Scope and objectives of the evaluation, main questions to be addressed Information sources and availability of information Methodological remarks, limitations encountered and validity of the findings

II. Countries and project background Brief countries context: an overview of the economy, the environment,

institutional development, demographic and other data of relevance to the project

Sector-specific issues of concern to the project 27 and important developments during the project implementation period

Project summary: o Fact sheet of the project: including project objectives and structure,

donors and counterparts, project timing and duration, project costs and co-financing

o Brief description including history and previous cooperation o Project implementation arrangements and implementation modalities,

institutions involved, major changes to project implementation o Positioning of the UNIDO project (other initiatives of government, other

donors, private sector, etc.) o Counterpart organization(s)

III. Project assessment This is the key chapter of the report and should address all evaluation criteria and questions outlined in the TOR (see section VI Project Evaluation Parameters). Assessment must be based on factual evidence collected and analyzed from different sources. The evaluators’ assessment can be broken into the following sections:

A. Design B. Relevance (Report on the relevance of project towards countries and

beneficiaries) C. Effectiveness (The extent to which the development intervention’s

objectives and deliverables were achieved, or are expected to be achieved, taking into account their relative importance)

27 Explicit and implicit assumptions in the logical framework of the project can provide insights into key-issues of concern (e.g. relevant legislation, enforcement capacities, government initiatives, etc.)

70

D. Efficiency (Report on the overall cost-benefit of the project and partner Countries contribution to the achievement of project objectives)

E. Sustainability of Project Outcomes (Report on the risks and vulnerability of the project, considering the likely effects of sociopolitical and institutional changes in partner countries, and its impact on continuation of benefits after the GEF project ends, specifically the financial, sociopolitical, institutional framework and governance, and environmental risks)

F. Assessment of monitoring and evaluation systems (Report on M&E design, M&E plan implementation, and Budgeting and funding for M&E activities)

G. Monitoring of long-term changes H. Assessment of processes affecting achievement of project results (Report

on preparation and readiness / quality at entry, country ownership, stakeholder involvement, financial planning, UNIDO support, cofinancing and project outcomes and sustainability, delays of project outcomes and sustainability, and implementation approach)

I. Project coordination and management (Report project management conditions and achievements, and partner countries commitment)

J. Gender mainstreaming K. Procurement issues

At the end of this chapter, an overall project achievement rating should be developed as required in Annex 2. The overall rating table required by the GEF should be presented here.

IV. Conclusions, Recommendations and Lessons Learned

This chapter can be divided into three sections:

A. Conclusions

This section should include a storyline of the main evaluation conclusions related to the project’s achievements and shortfalls. It is important to avoid providing a summary based on each and every evaluation criterion. The main conclusions should be cross-referenced to relevant sections of the evaluation report.

B. Recommendations

This section should be succinct and contain few key recommendations. They should:

be based on evaluation findings realistic and feasible within a project context indicate institution(s) responsible for implementation (addressed to a

specific officer, group or entity who can act on it) and have a proposed timeline for implementation if possible

be commensurate with the available capacities of project team and partners

take resource requirements into account.

71

Recommendations should be structured by addressees:

o UNIDO o Government and/or Counterpart Organizations o Donor

C. Lessons Learned

Lessons learned must be of wider applicability beyond the evaluated project but must be based on findings and conclusions of the evaluation

For each lesson the context from which they are derived should be briefly stated

Annexes should include the evaluation TOR, list of interviewees, documents reviewed, a summary of project identification and financial data, and other detailed quantitative information. Dissident views or management responses to the evaluation findings may later be appended in an annex.

11.2 Annex 2 - Overall Ratings Table

Criterion Evaluator’s Summary Comments

Evaluator’s Rating

Attainment of project objectives and results (overall rating)

Sub criteria (below)

Effectiveness

Relevance

Efficiency

Sustainability of Project outcomes (overall rating) Sub criteria (below)

Financial risks

Sociopolitical risks

Institutional framework and governance risks

Environmental risks

72

Criterion Evaluator’s Summary Comments

Evaluator’s Rating

Monitoring and Evaluation (overall rating) Sub criteria (below)

M&E Design

M&E Plan Implementation (use for adaptive management)

Budgeting and Funding for M&E activities

UNIDO specific ratings

Quality at entry / Preparation and readiness

Implementation approach

UNIDO Supervision and backstopping

Overall Rating

RATING OF PROJECT OBJECTIVES AND RESULTS

• Highly Satisfactory (HS): The project had no shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

• Satisfactory (S): The project had minor shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

• Moderately Satisfactory (MS): The project had moderate shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

• Moderately Unsatisfactory (MU): The project had significant shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

• Unsatisfactory (U) The project had major shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

• Highly Unsatisfactory (HU): The project had severe shortcomings in the achievement of its objectives, in terms of relevance, effectiveness or efficiency.

Please note: Relevance and effectiveness will be considered as critical criteria. The overall rating of the project for achievement of objectives and results may not be higher than the lowest rating on either of these two criteria. Thus, to have an overall satisfactory rating for outcomes a project must have at least satisfactory ratings on both relevance and effectiveness.

RATINGS ON SUSTAINABILITY

73

Sustainability will be understood as the probability of continued long-term outcomes and impacts after the GEF project funding ends. The evaluation will identify and assess the key conditions or factors that are likely to contribute or undermine the persistence of benefits beyond project completion. Some of these factors might be outcomes of the project, i.e. stronger institutional capacities, legal frameworks, socio-economic incentives /or public awareness. Other factors will include contextual circumstances or developments that are not outcomes of the project but that are relevant to the sustainability of outcomes.

Rating system for sustainability sub-criteria

On each of the dimensions of sustainability of the project outcomes will be rated as follows.

• Likely (L): There are no risks affecting this dimension of sustainability.

• Moderately Likely (ML). There are moderate risks that affect this dimension of sustainability.

• Moderately Unlikely (MU): There are significant risks that affect this dimension of sustainability.

• Unlikely (U): There are severe risks that affect this dimension of sustainability.

All the risk dimensions of sustainability are critical. Therefore, overall rating for sustainability will not be higher than the rating of the dimension with lowest ratings. For example, if a project has an Unlikely rating in either of the dimensions then its overall rating cannot be higher than Unlikely, regardless of whether higher ratings in other dimensions of sustainability produce a higher average.

RATINGS OF PROJECT M&E

Monitoring is a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing project with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds. Evaluation is the systematic and objective assessment of an on-going or completed project, its design, implementation and results. Project evaluation may involve the definition of appropriate standards, the examination of performance against those standards, and an assessment of actual and expected results.

The Project monitoring and evaluation system will be rated on ‘M&E Design’, ‘M&E Plan Implementation’ and ‘Budgeting and Funding for M&E activities’ as follows:

• Highly Satisfactory (HS): There were no shortcomings in the project M&E system. • Satisfactory(S): There were minor shortcomings in the project M&E system. • Moderately Satisfactory (MS): There were moderate shortcomings in the project

M&E system. • Moderately Unsatisfactory (MU): There were significant shortcomings in the project

M&E system. • Unsatisfactory (U): There were major shortcomings in the project M&E system. • Highly Unsatisfactory (HU): The Project had no M&E system.

74

“M&E plan implementation” will be considered a critical parameter for the overall assessment of the M&E system. The overall rating for the M&E systems will not be higher than the rating on “M&E plan implementation.”

All other ratings will be on the GEF six point scale:

HS = Highly Satisfactory Excellent

S = Satisfactory Well above average

MS = Moderately Satisfactory Average

MU = Moderately Unsatisfactory Below Average

U = Unsatisfactory Poor

HU = Highly Unsatisfactory Very poor (Appalling)

11.3 Annex 3 - GEF Minimum Requirements for M&E28

Minimum Requirement 1: Project Design of M&E All projects will include a concrete and fully budgeted monitoring and evaluation plan by the time of work program entry for full-sized projects and CEO approval for medium-sized projects. This monitoring and evaluation plan will contain as a minimum:

• SMART indicators for project implementation, or, if no indicators are identified, an alternative plan for monitoring that will deliver reliable and valid information to management;

• SMART indicators for results (outcomes and, if applicable, impacts), and, where appropriate, indicators identified at the corporate level;

• baseline for the project, with a description of the problem to be addressed, with indicator data, or, if major baseline indicators are not identified, an alternative plan for addressing this within one year of implementation;

• identification of reviews and evaluations that will be undertaken, such as mid-term reviews or evaluations of activities; and

28 http://www.thegef.org/gef/sites/thegef.org/files/documents/ME_Policy_2010.pdf

75

• organizational set-up and budgets for monitoring and evaluation.

Minimum Requirement 2: Application of Project M&E Project monitoring and supervision will include implementation of the M&E plan, comprising:

• SMART indicators for implementation are actively used, or if not, a reasonable explanation is provided;

• SMART indicators for results are actively used, or if not, a reasonable explanation is provided;

• the baseline for the project is fully established and data compiled to review progress reviews, and evaluations are undertaken as planned; and

• the organizational set-up for M&E is operational and budgets are spent as planned.

11.4 Annex 4 – Required Project Identification and Financial Data

The evaluation report should provide information on project identification, time frame, actual expenditures, and co-financing in the following format, which is modeled after the project identification form (PIF).

I. Project general information:

Project Title

GEF ID Number

UNIDO ID (SAP Number)

Region

Country(ies)

GEF Focal Area and Operational Program:

Co-Implementing Agency(ies)

GEF Agencies (Implementing Agency)

Project Executing Partners

76

Project Size (FSP, MSP, EA)

Project CEO Endorsement/Approval Date

Project Implementation Start Date (PAD Issuance Date)

Original Expected Implementation End Date (indicated in CEO Endorsement/Approval document)

Revised Expected Implementation End Date (if any)

Project Duration (Months)

GEF Grant (USD)

GEF PPG (USD) (if any)

Co-financing (USD) at CEO Endorsement

Total Project Cost (USD) (GEF Grant + Co-financing at CEO Endorsement)

Agency Fee (USD)

II. Dates

Milestone Expected Date Actual Date

Project CEO Endorsement/Approval Date

Project Implementation Start Date (PAD Issuance Date)

Original Expected Implementation End Date (indicated in CEO Endorsement/Approval document)

Revised Expected Implementation End Date (if any)

Mid-term evaluation completion

Planned Tracking Tool Date

77

III. Project Framework

Project Component Activity

Type

GEF Financing (in $) Cofinancing (in $)

Approved Actual Promised Actual

1.

2.

3.

4.

5.

6. Project Management

Total

Activity types are:

Experts, researches hired technical assistance, Workshop, Meetings or experts

consultation scientific and technical analysis, experts researches hired

Promised co-financing refers to the amount indicated on endorsement/approval.

IV. Co-financing

Project preparation Project implementation

Total

Source of cofinancing

Type Expected Actual Expected Actual Expected Actual

Host gov’t contribution

78

GEF Agency (ies)

Bilateral aid agency (ies)

Multilateral agency (ies)

Private sector

NGO

Other

Total cofinancing

Expected amounts are those submitted by the GEF Agencies in the original project appraisal document. Co-financing types are grant, soft loan, hard loan, guarantee, in kind, or cash.

79

11.5 Annex 5. Job Descriptions


TERMS OF REFERENCE FOR PERSONNEL UNDER INDIVIDUAL SERVICE AGREEMENT (ISA) –

Independent Evaluation of UNIDO project: “Environmentally Sound Management and Final Disposal of PCBs in India”

GFIND10001 SAP ID: 104044

Title: Senior International Evaluation Consultant

Main Duty Station and Location: Home-based

Mission/s to: Vienna, Austria and Delhi, India

Start of Contract (EOD): 01 September 2014

End of Contract (COB): 31 October 2014

Number of Working Days: 28

ORGANIZATIONAL CONTEXT

The Office for Independent Evaluation is responsible for the independent evaluation function of UNIDO. It supports learning, continuous improvement and accountability, and provides factual information about result and practices that feed into the programmatic and strategic decision-making processes.

PROJECT CONTEXT See evaluation terms of reference (attached).

The senior international evaluation consultant will act as a Team leader in this UNIDO/GEF Mid-Term Project Evaluation according to the terms of reference. She/he will be responsible for the preparation of the evaluation report, including the coordination of inputs from the National Evaluation Consultant. This concerns in

80

particular the overall assessment of evaluation issues in section VI of the TOR. The Team Leader will perform the following tasks:

Main duties Duration/ location

Deliverables

Review project documentation and relevant country background information (national policies and strategies, UN strategies and general economic data…); determine key data to collect in the field and prepare key instruments (questionnaires, logic models…) to collect these data through interviews and/or surveys during and prior to the field missions

Assess the adequacy of the POPs legislative and regulatory framework for India

5 days

Home based

Draft inception report, including list of detailed evaluation questions; questionnaires/ interview guidelines; logic models; list of key data to collect, draft list of stakeholders to interview during the field missions

Brief assessment of the adequacy of the country’s legislative and regulatory framework

Briefing at HQs 2 days

Vienna, Austria

All necessary documents and information from UNIDO Project Manager gathered

Conduct field mission to India in September 2014

7 days

(including travel days)

India

Presentations of the evaluation’s initial findings, draft conclusions and recommendations to stakeholders in India at the end of the missions

Agreement with the National Consultant on the structure and content of the evaluation report and the distribution of writing tasks

Present preliminary findings and recommendations to the stakeholders at UNIDO HQ (incl. travel)

2 days

Vienna, Austria

Presentation slides

Prepare the evaluation report according to TOR and template provided by UNIDO EVA

Coordinate the inputs from the National Consultant and combine with her/his own inputs into the final

8 days

Home based

Draft evaluation report

81


Deliverables

draft evaluation report

Revise the draft project evaluation reports based on comments from UNIDO Office for Independent Evaluation and stakeholders and edit the language and form the final version according to UNIDO standards

4 days

Home based

Final evaluation report

TOTAL 28 days

REQUIRED COMPETENCIES

Core values: 1. Integrity 2. Professionalism 3. Respect for diversity Core competencies: 1. Results orientation and accountability 2. Planning and organizing 3. Communication and trust 4. Team orientation 5. Client orientation 6. Organizational development and innovation Managerial competencies (as applicable): 1. Strategy and direction 2. Managing people and performance 3. Judgement and decision making 4. Conflict resolution

MINIMUM ORGANIZATIONAL REQUIREMENTS Education: Advanced university degree in environment, energy, engineering, economics, development studies or other relevant discipline with a specialization in Stockholm Convention issues.

Technical and Functional Experience:

A minimum of twenty years practical experience in the field of environmental projects design, evaluation and monitoring, including experience at the international level involving technical cooperation in developing countries is required. Exposure to the needs, conditions and problems in developing countries.

82

Languages: Fluency in written and spoken English is required. Fluency and/or working knowledge of another official UN language, particularly French desirable.

Absence of Conflict of Interest: According to UNIDO rules, the consultant must not have been involved in the design and/or implementation, supervision and coordination of and/or have benefited from the programme/project (or theme) under evaluation. The consultant will be requested to sign a declaration that none of the above situations exists and that the consultants will not seek assignments with the manager/s in charge of the project before the completion of her/his contract with the Office for Independent Evaluation.


TERMS OF REFERENCE FOR PERSONNEL UNDER INDIVIDUAL SERVICE AGREEMENT (ISA) –

11.5.1 Independent Evaluation of UNIDO project: “Environmentally Sound Management and Final Disposal of PCBs in India”

11.5.2 GFIND10001 SAP ID: 104044

Title: National Evaluation Consultant

Main Duty Station and Location: Home-based

Mission/s to: Travel within India

Start of Contract (EOD): 01 September 2014

End of Contract (COB): 31 October 2014

Number of Working Days: 21

11.5.2.1.1.1.1 ORGANIZATIONAL CONTEXT

The Office for Independent Evaluation is responsible for the independent evaluation function of UNIDO. It supports learning, continuous improvement and accountability,

83

and provides factual information about result and practices that feed into the programmatic and strategic decision-making processes.

11.5.2.1.1.1.2 PROJECT CONTEXT See evaluation terms of reference (attached).

The national evaluation consultant will act as a team member in this UNIDO/GEF mid-term project evaluation according to the terms of reference. She/he will participate in the preparation of the evaluation report, including the coordination of inputs with the team leader - international evaluation consultant. This concerns in particular the overall assessment of evaluation issues in section VI of the TOR. The national evaluation eonsultant will perform the following tasks:


Deliverables

Review project documentation and relevant country background information (national policies and strategies, UN strategies and general economic data…)

Coordinate with the National Proejct Coordinator the planning of the evaluation field mission and contacting concerned organizations to prepare the evaluation programme

5 days

Home based

Inputs, feedback and comments to the inception report

Evaluation mission programme

Carry out meetings, visits and interviews of stakeholders according to the evaluation programme and facilitate the work of the evaluation team in India (including acting as interpreter)

Participate in drafting the main conclusions and recommendations, and present them to stakeholders in accordance with the instructions of the International Evaluation Consultant

8 days

Field mission, travel within India

Notes, tables; information gathered on issues specified in ToR

Draft conclusions and recommendations to stakeholders

Contribute to the draft report as assigned by the International Evaluation Consultant

5 days

Home based

First draft of chapters on the country background and other inputs into the draft evaluation report as agreed with the International Evaluation

84


Deliverables

Consultant

Revise the draft chapters based on comments from UNIDO Office for Independent Evaluation and stakeholders

3 days

Home based

Final evaluation report

TOTAL 21 days

11.5.2.1.1.1.3 REQUIRED COMPETENCIES

Core values: 1. Integrity 2. Professionalism 3. Respect for diversity Core competencies: 1. Results orientation and accountability 2. Planning and organizing 3. Communication and trust 4. Team orientation 5. Client orientation 6. Organizational development and innovation Managerial competencies (as applicable): 1. Strategy and direction 2. Managing people and performance 3. Judgement and decision making 4. Conflict resolution

11.5.2.1.2 MINIMUM ORGANIZATIONAL REQUIREMENTS Education: Advanced university degree in environment, energy, engineering, economics, development studies or other relevant discipline.

Technical and Functional Experience:

A minimum of ten years practical experience in the field of evaluation and monitoring, including experience at the international level involving technical cooperation in developing countries. Exposure to the needs, conditions and problems in developing countries.

85

Languages: Fluency in written and spoken English and Hindi is required.

Absence of Conflict of Interest: According to UNIDO rules, the consultant must not have been involved in the design and/or implementation, supervision and coordination of and/or have benefited from the programme/project (or theme) under evaluation. The consultant will be requested to sign a declaration that none of the above situations exists and that the consultants will not seek assignments with the manager/s in charge of the project before the completion of her/his contract with the Office for Independent Evaluation.

11.6 Annex 6 – UNIDO Procurement Process

UNIDO Procurement Process -- Generic Approach and Assessment Framework –

1. Introduction This document outlines an approach and encompasses a framework for the assessment of UNIDO procurement processes, to be included as part of country evaluations as well as in technical cooperation (TC) projects/programmes evaluations.

The procurement process assessment will review in a systematic manner the various aspects and stages of the procurement process being a key aspect of the technical cooperation (TC) delivery. These reviews aim to diagnose and identify areas of strength as well as where there is a need for improvement and lessons.

The framework will also serve as the basis for the “thematic evaluation of the procurement process efficiency” to be conducted in 2015 as part of the ODG/EVA work programme for 2014-15.

2. Background Procurement is defined as the overall process of acquiring goods, works, and services, and includes all related functions such as planning, forecasting, supply chain management, identification of needs, sourcing and solicitation of offers, preparation and award of contract, as well as contract administration until the final discharge of all obligations as defined in the relevant contract(s). The procurement process covers activities necessary for the purchase, rental, lease or sale of goods, services, and other requirements such as works and property.

Past project and country evaluations commissioned by ODG/EVA raised several issues related to procurement and often efficiency related issues. It also became obvious that there is a shared responsibility in the different stages of the procurement

86

process which includes UNIDO staff, such as project managers, and staff of the procurement unit, government counterparts, suppliers, local partner agencies (i.e. UNDP), customs and transport agencies etc..

In July 2013, a new “UNIDO Procurement Manual” was introduced. This Procurement Manual provides principles, guidance and procedures for the Organization to attain specified standards in the procurement process. The Procurement Manual also establishes that “The principles of fairness, transparency, integrity, economy, efficiency and effectiveness must be applied for all procurement transactions, to be delivered with a high level of professionalism thus justifying UNIDO’s involvement in and adding value to the implementation process”.

To reduce the risk of error, waste or wrongful acts and the risk of not detecting such problems, no single individual or team controls shall control all key stages of a transaction. Duties and responsibilities shall be assigned systemically to a number of individuals to ensure that effective checks and balances are in place.

In UNIDO, authorities, responsibilities and duties are segregated where incompatible. Related duties shall be subject to regular review and monitoring. Discrepancies, deviations and exceptions are properly regulated in the Financial Regulations and Rules and the Staff Regulations and Rules. Clear segregation of duties is maintained between programme/project management, procurement and supply chain management, risk management, financial management and accounting as well as auditing and internal oversight. Therefore, segregation of duties is an important basic principle of internal control and must be observed throughout the procurement process.

The different stages of the procurement process should be carried out, to the extent possible, by separate officials with the relevant competencies. As a minimum, two officials shall be involved in carrying out the procurement process. The functions are segregated among the officials belonging to the following functions:

• Procurement Services: For carrying out centralized procurement, including review of technical specifications, terms of reference, and scope of works, market research/surveys, sourcing/solicitation, commercial evaluation of offers, contract award, contract management;

• Substantive Office: For initiating procurement requests on the basis of well formulated technical specifications, terms of reference, scope of works, ensuring availability of funds, technical evaluation of offers; award recommendation; receipt of goods/services; supplier performance evaluation. In respect of decentralized procurement, the segregation of roles occur between the Project Manager/Allotment Holder and his/her respective Line Manager. For Fast Track procurement, the segregate on occurs between the Project Manager/Allotment Holder and Financial Services;

• Financial Services: For processing payments.

Figure 1 presents a preliminary “Procurement Process Map”, showing the main stages, stakeholders and their respective roles and responsibilities. During 2014/2015, in preparation for the thematic evaluation of the procurement process in 2015, this process map/ workflow will be further refined and reviewed.

87

Figure 1: UNIDO Procurement Process Map

3. Purpose The purpose of the procurement process assessments is to diagnose and identify areas for possible improvement and to increase UNIDO’s learning about strengths and weaknesses in the procurement process. It will also include an assessment of the adequacy of the ‘Procurement Manual” as a guiding document.

The review is intended to be useful to managers and staff at UNIDO headquarters and in the field offices (project managers, procurement officers), who are the direct involved in procurement and to UNIDO management.

4. Scope and focus

Procurement process assessments will focus on the efficiency aspects of the procurement process, and hence it will mainly fall under the efficiency evaluation criterion. However, other criteria such as effectiveness will also be considered as needed.

88

These assessments are expected to be mainstreamed in all UNIDO country and project evaluations to the extent of its applicability in terms of inclusion of relevant procurement related budgets and activities.

A generic evaluation matrix has been developed and is found in Annex B. However questions should be customized for individual projects when needed.

5. Key Issues and Evaluation Questions Past evaluations and preliminary consultations have highlighted the following aspects or identified the following issues:

- Timeliness. Delays in the delivery of items to end-users. - Bottlenecks. Points in the process where the process stops or considerably

slows down. - Procurement manual introduced, but still missing subsidiary templates and

tools for its proper implementation and full use. - Heavy workload of the procurement unit and limited resources and increasing

“procurement demand” - Lack of resources for initiating improvement and innovative approaches to

procurement (such as Value for Money instead of lowest price only, Sustainable product lifecycle, environmental friendly procurement, etc.)

- The absence of efficiency parameters (procurement KPIs)

On this basis, the following evaluation questions have been developed and would be included as applicable in all project and country evaluations in 2014-2015

- To what extent does the process provide adequate treatment to different types of procurement (e.g. by value, by category, by exception…)

- Was the procurement timely? How long the procurement process takes (e.g. by value, by category, by exception…)

- Did the good/item(s) arrive as planned or scheduled? If no, how long were the times gained or delays. If delay, what was the reason(s)?

- Were the procured good(s) acquired at a reasonable price?

- To what extent were the procured goods of the expected/needed quality and quantity?

- Were the transportation costs reasonable and within budget. If no, pleased elaborate.

- Was the freight forwarding timely and within budget?. If no, pleased elaborate.

- Who was responsible for the customs clearance? UNIDO FO? UNDP? Government? Other?

89

- Was the customs clearance handled professionally and in a timely manner? How many days did it take?

- How long time did it take to get approval from the government on import duty exemption?

- Which were the main bottlenecks / issues in the procurement process?

- Which good practices have been identified?

- To what extent roles and responsibilities of the different stakeholders in the different procurement stages are established, adequate and clear?

- To what extent there is an adequate segregation of duties across the procurement process and between the different roles and stakeholders?

6. Evaluation Method and Tools These assessments will be based on a participatory approach, involving all relevant stakeholders (e.g. process owners, process users and clients).

The evaluation tools to be considered for use during the reviews are:

- Desk Review: Policy, Manuals and procedures related to the procurement process. Identification of new approaches being implemented in other UN or international organizations. Findings, recommendations and lessons from UNIDO Evaluation reports.

- Interviews: to analyze and discuss specific issues/topics with key process stakeholders

- Survey to stakeholders: To measure the satisfaction level and collect expectations, issues from process owners, user and clients

- Process and Stakeholders Mapping: To understand and identify the main phases the procurement process and sub-processes; and to identify the perspectives and expectations from the different stakeholders, as well as their respective roles and responsibilities

- Historical Data analysis from IT procurement systems: To collect empirical data and identify and measure to the extent possible different performance dimensions of the process, such as timeliness, re-works, complaints, ..)

An evaluation matrix is presented in Annex A, presenting the main questions and data sources to be used in the project and country evaluations, as well as the preliminary questions and data sources for the forthcoming thematic evaluation on Procurement process in 2015.

90

ANNEX A: Evaluation Matrix for the Procurement Process

No. Area Evaluation

Question Indicators29

Data Source(s) For Country / Project Evaluations

Additional data Source(s) For Thematic Evaluation of procurement process in 2015.

Timeliness

- Was the procurement timely? How long the procurement process takes (e.g. by value, by category, by exception…)

(Overall) Time to Procure (TTP)

• Interviews with PMs, Government counterparts and beneficiaries

• Procurement related documents review

• SAP/Infobase (queries related to procurement volumes, categories, timing, issues)

• Evaluation Reports

• Survey to PMs, procurement officers, beneficiaries, field local partners.

• Interviews with Procurement officers

- Did the good/item(s) arrive as planned or scheduled? If no, how long were the times gained or delays. If delay, what was the reason(s)?

Time to Delivery (TTD)

• Interviews with PM, procurement officers and Beneficiaries

- Was the freight forwarding timely and within budget? If

29 These indicators are preliminary proposed here. They will be further defined and piloted during the Thematic Evaluation of UNIDO procurement process planned for 2015.

No. Area Evaluation




no, pleased elaborate.

- Was the customs clearance timely? How many days did it take?


- How long time did it take to get approval from the government on import duty exemption

Time to Government Clearance (TTGC)

• Interviews with beneficiaries

Roles and Responsibilities

- To what extent roles and responsibilities of the different stakeholders in the different procurement stages are established, adequate and clear?

Level of clarity of roles and responsibilities

• Procurement Manual

• Interview with PMs





- To what extent there is an


• Interview

92

No. Area Evaluation




adequate segregation of duties across the procurement process and between the different roles and stakeholders?

with PMs

- How was responsibility for the customs clearance arranged? UNIDO FO? UNDP? Government? Other?


• Interview to PMs

• Interviews with local partners

- To what extent were suppliers delivering products/ services as required?

Level of satisfaction with Suppliers

• Interviews with PMs

Costs

- Were the transportation costs reasonable and within budget. If no, pleased

• Interviews with PMs


• Survey to PMs, procurement officers, beneficiarie

93

No. Area Evaluation




elaborate. s, field local partners.


- Were the procured goods/services within the expected/planned costs? If no, please elaborate

Costs vs budget


Quality of Products

- To what extent the process provides adequate treatment to different types of procurement (e.g. by value, by category, by exception…)





- To what extent were the procured goods of the expected/needed quality and quantity?.

Level of satisfaction with products/services

• Survey to PMs and beneficiaries

• Observation in project site

Process / workflow

- To what extent the procurement process if fit for purpose?

Level of satisfaction with the procurement process

• Interviews with PMs, Government counterparts and


• Evaluation

94

No. Area Evaluation




beneficiaries

Reports • Survey to

PMs, procurement officers, beneficiaries, field local partners.





- Which are the main bottlenecks / issues in the procurement process?


- Which part(s) of the procurement process can be streamlined or simplified?


95

Documents

ENVIRONMENTALLY SOUND MANAGEMENT AND FINAL … · The Steel Authority of India (SAIL)’s Bhilai Steel Plant (BSP), is the lead beneficiary of this project. SAIL, Under the Ministry