Document of The World Bank€¦ · 300 (2012) 600 (2014) 933 Date 12/30/2003 12/15/2012 06/12/2015 06/12/2015 Comments Surpassed (155% achieved). Targets were revised upwards in the

Document of The World Bank

Report No.: ICR00003532

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-74450)

ON A LOAN

IN THE AMOUNT OF US$26 MILLION

TO THE

ORIENTAL REPUBLIC OF URUGUAY

FOR A

PROMOTING INNOVATION TO ENHANCE COMPETITIVENESS PROJECT

December 10, 2015

Education Global Practice Latin America and the Caribbean Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

i

CURRENCY EQUIVALENTS (Exchange Rate Effective November 18, 2015)

Currency Unit = Uruguayan Peso

UYP$1.00 = US$0.03 US$1.00 = UYP$29.52

FISCAL YEAR January 1 – December 31

ABBREVIATIONS AND ACRONYMS

ANII National Research and Innovation Agency (Agencia Nacional de Investigación e Innovación)

CND National Development Corporation DINACYT

National Directorate of Science, Technology, and Innovation (Dirección Nacional de Ciencia, Tecnología e Innovación)

FDI Foreign Direct Investment FM Financial Management GII Global Innovation Index GMI Inter-Ministerial Innovation Cabinet (Gabinete Ministerial de la Innovación) IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report IDA International Development Association IDB Inter-American Development Bank INIA National Institute for Agricultural Research (Instituto Nacional de Investigación

Agropecuaria) IRR Internal Rate of Return LATU Technological Laboratory of Uruguay (Laboratorio Tecnológico del Uruguay) M&E Monitoring and Evaluation MEC Ministry of Education and Culture (Ministerio de Educación y Cultura) MEF Ministry of Economy and Finance (Ministerio de Economía y Finanzas) OECD Organisation for Economic Co-operation and Development PAD Project Appraisal Document PDO Project Development Objective PENCTI National Strategic Plan for Science, Technology, and Innovation (Plan

Estratégico Nacional en Ciencia, Tecnología e Innovación) PPF Project Preparation Facility R&D Research and Development RICYT Network on Science and Technology Indicators - Ibero-American and Inter-

American (Red de Indicadores de Ciencia y Tecnología - Iberoamericana e Interamericana)

SCI Science Citation Index SME Small and Medium Enterprise SNI National Researchers System (Sistema Nacional de Investigadores) STI Science, Technology, and Innovation

ii

TIMBO Internet portal offering access to multiple sources of science publications (Trama Interinstitucional y Multidisciplinaria de Bibliografía On-line)

UdelaR University of the Republic (Universidad de la República) UTU Uruguayan University of Labor (Universidad del Trabajo del Uruguay) WDI World Development Indicators

Senior Global Practice Director: Claudia Maria Costin

Practice Manager: Reema Nayar

Project Team Leader: Diego Ambasz

ICR Team Leader: Diego Ambasz

ICR Primary Author: Sara Troiano

iii

URUGUAY Promoting Innovation to Enhance Competitiveness Project

CONTENTS

Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design .......................................................... 1

2. Key Factors Affecting Implementation and Outcomes ......................................................... 4

3. Assessment of Outcomes ........................................................................................................ 10

4. Assessment of Risk to Development Outcome ..................................................................... 20

5. Assessment of Bank and Borrower Performance ................................................................ 21

6. Lessons Learned ...................................................................................................................... 22

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ................... 24

Annex 1. Project Costs and Financing ...................................................................................... 25

Annex 2. Outputs by Component .............................................................................................. 26

Annex 3. Economic and Financial Analysis ............................................................................. 35

Annex 4. Bank Lending and Implementation Support/Supervision Processes ..................... 42

Annex 5. Beneficiary Survey Results ........................................................................................ 44

Annex 6. Stakeholder Workshop Report and Results ............................................................. 45

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ............................. 46

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ............................... 56

Annex 9. List of Supporting Documents ................................................................................... 57

Annex 10. The Concepts of Transfer and Adaptation of Knowledge and Technology ........ 59

MAP ............................................................................................................................................. 60

iv

A. Basic Information

Country: Uruguay Project Name: UY Promoting Innovation to Enhance Competitiveness

Project ID: P095520 L/C/TF Number(s): IBRD-74450

ICR Date: 11/18/2015 ICR Type: Core ICR

Lending Instrument: Specific Investment Loan Borrower: Government of the Oriental Republic of Uruguay

Original Total Commitment:

US$26.00 million Disbursed Amount: US$26.00 million

Revised Amount: US$26.00 million

Environmental Category: C

Implementing Agencies: Agencia Nacional de Investigación e Innovación (National Research and Innovation Agency)

Cofinanciers and Other External Partners: B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 06/01/2006 Effectiveness: 08/06/2007 08/06/2007

Appraisal: 12/13/2006 Restructuring(s):

01/22/2008 12/01/2008 06/25/2010 12/12/2012 08/14/2014

Approval: 05/01/2007 Mid-term Review: 11/29/2010 12/03/2010

Closing: 12/15/2012 06/12/2015 C. Ratings Summary C.1 Performance Rating by ICR

Outcomes: Satisfactory

Risk to Development Outcome: Low or Negligible

Bank Performance: Moderately Satisfactory

Borrower Performance: Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)

Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Satisfactory

v

Quality of Supervision: Moderately Satisfactory Implementing Agency/Agencies:

Satisfactory

Overall Bank Performance:

Moderately Satisfactory Overall Borrower Performance:

Satisfactory

C.3 Quality at Entry and Implementation Performance Indicators

Implementation Performance

Indicators QAG Assessments (if

any) Rating

Potential Problem Project at any time (Yes/No):

No Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

No Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:

Moderately Satisfactory

D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

Central government administration 11 11

General industry and trade sector 42 42

Tertiary education 47 47

Theme Code (as % of total Bank financing)

Education for the knowledge economy 40 40

Export development and competitiveness 20 20

Technology diffusion 40 40 E. Bank Staff

Positions At ICR At Approval

Vice President: Jorge Familiar Pamela Cox

Country Director: Jesko Hentschel Axel van Trotsenburg

Practice Manager/Manager: Reema Nayar Eduardo Velez Bustillo (LCSHE) and Aquiles Almansi (LCSPF)

Project Team Leader: Diego Ambasz Kristian Thorn (LCSHE) and Esperanza Lasagabaster (LCSFP) (Co-Team Leaders)

ICR Team Leader: Diego Ambasz

ICR Primary Author: Sara Troiano

vi

F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document) The project development objective (PDO) is to strengthen Uruguay's capacity to generate, transfer, and adapt knowledge and technology by supporting: (i) the institutional framework for science, technology, and innovation (STI); (ii) investments in human capital and high-quality research teams; (iii) technology transfer and private sector innovation; and (iv) cross-sectoral and international research collaboration and mobility of researchers between public institutions and the productive sector. Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1

Proportion of public research and development (R&D) funding administered by the National Research and Innovation Agency (Agencia Nacional de Investigación e Innovación, ANII) (excluding University of the Republic [Universidad de la República, UdelaR]) (annual)

Value 0% 40% 10.4%

Date 12/13/2006 12/15/2012 06/12/2015

Comments

Dropped. Indicator dropped in the 2012 restructuring. The 2012 review of the Project has assessed it as an input indicator rather than as an outcome as it does not refer directly to the Project objectives. Public R&D funding includes full financing of public research institutes (for example, the Technological Laboratory of Uruguay [Laboratorio Tecnológico del Uruguay, LATU] and the National Institute of Research in Agriculture and Livestock [Instituto Nacional de Investigación Agropecuaria, INIA]), as well as national programs aimed at stimulating education (for example, Plan Ceibal - One Laptop per Child). Given the infrastructure and other fixed costs associated with these institutes and programs, they account for a large part of total public spending in R&D. On the contrary, fixed costs associated with the ANII are minimal, and most of public funding administered by the ANII is directly channeled to finance R&D subprojects. The ANII is, for example, the main public institution in the country in terms of innovation grants to the private sector.

Indicator 2 Number of research groups with more than 3 researchers registered in the national researcher classification system (cumulative)

Value 0 50 N/A

Date 12/13/2006 12/15/2012

Comments

Dropped. Indicator dropped in the 2010 restructuring. It is not possible to measure the indicator as designed because the National Researcher System (Sistema Nacional de Investigadores, SNI) does not register groups but only individuals. As of 2015, there are 1,400 individual researchers registered in the SNI, against the 1001 researchers registered at the moment of its creation (2009).

vii

Indicator 3 Number of internationally published scientific articles

Value 653 (418 with original data source)

708 (462 with original data source)

1,000 (cumulative, 2012) 1,000 (annually, 2014)

1,263 (annual, 2014) (cumulative: 6,835)

Date 12/30/2003 12/15/2012 06/14/2014 12/30/2014

Comments

Surpassed (126% achieved). In the 2010 restructuring, the original data source used to track this indicator, the Science Citation Index (SCI) international publications database (Thomson Reuters) was replaced by the Scopus database (Elsevier). Baselines and targets were changed accordingly. The target was revised upwards in the 2012 restructuring to reflect progress to date and the extended implementation period. In 2014, the indicator was further revised to measure articles published annually as opposed to the cumulative frequency envisaged in the Project Appraisal Document (PAD). The target remained unchanged despite the fact that the change of frequency made it more difficult to achieve.

Indicator 4 Number of graduates from domestic master’s and PhD programs (annual)

Value 179 240 300 (2012) 600 (2014)

933

Date 12/30/2003 12/15/2012 06/12/2015 06/12/2015

Comments

Surpassed (155% achieved). Targets were revised upwards in the 2012 and 2014 restructurings to reflect progress to date and the extended implementation period. Last data available are from 2014 since it takes 12 months for the Ministry of Education and Culture to collect and publish this information.

Indicator 5 Number of project-supported researchers inserted into industry (cumulative)

Value 0 40 10 (2010) 25 (2012) 30 (2014)

44

Date 12/13/2006 12/15/2012 06/12/2015 06/12/2015

Comments

Surpassed (147% achieved). The target for this indicator was changed three times. In 2010, it was scaled down to reflect the closing of a special type of scholarship financed by the ANII (supporting internships in private companies), which was assumed to have a direct impact on the number of supported researchers inserted into industry. In 2012 and 2014, the target was eventually revised upward to reflect the better-than-expected results to date as well as the extended implementation period. Examples of the type of work that led researchers to find a job in the private sector include assessment of the economic potential of wind energy in Uruguay, smart grid development, creation on new optical methods for monitoring the atmosphere (Engineering); development of new instruments to diagnose and assess patients of cystic fibrosis; study on the effectiveness of anti-cancer pharmaceuticals based on metallic molecules (Medicine); safety and health conditions in the forestry industry in Uruguay (Sociology); evaluation of the TACCH methodology to cure pervasive developmental disorders (Psychology); and elasticity of electric consumption among households in Uruguay (Economics).

Indicator 6 Proportion of variable costs of supported technology centers generated by the sale of services (annual %)

Value 0% 40% N/A

viii

Date 12/13/2006 12/15/2012

Comments

Dropped. Indicator dropped in the 2010 restructuring to reflect a change in the Project strategy. The initial plan was to set up Technology Centers offering services to private companies. As of 2010, that approach did not work due to low demand, and instead the Project focused on direct support of Alliances and Networks (more flexible instruments). Consequently, indicators related to Technology Centers were dropped. In 2012, recognizing new favorable conditions for the development of Centers, the ANII renewed efforts to promote this activity. The instrument was officially launched to the public in 2013, but the approved Centers are still in a too early stage of development to be able to measure this indicator.

Indicator 7 Number of patent applications submitted by supported research consortia (cumulative)

Value 0 4 1

Date 12/13/2006 12/15/2012 06/12/2015

Comments

Dropped. Indicator dropped in the 2010 restructuring. Uruguay is not a member of the Patents Cooperation Treaty (PCT), which makes it difficult and less meaningful for national researchers/companies to apply for a patent which would be valid only in Uruguay. Under the current activities of public-private partnerships (Component 3), 1 patent application was submitted (preclinical evaluation of probiotic strains for dairy products, developed by one Alliance). There are several reasons why patents do not represent a good indicator of innovation outcome in Uruguay. First, most innovations do not lead to patents since they are more about the adoption of technologies new to the firms or to the country but not necessarily new to the world or about process, organizational, or services innovation that do not lead to patents, by nature. This is even more true in developing economies, where patents are very rare and materialize only in some top and consolidated research centers. Besides, patents can be at the very end of an innovative process, which may happen some years after the beginning of a research or the creation of a public-private partnership. Finally, it must be noted that in Uruguay many representatives from the private sector, including those benefitting from the ANII's instruments under Component 3, expressed reluctance to patent to better guarantee the secrecy (paradoxically, patenting was seen as a way to lose protection) and to avoid dealing with a patents system that is costly at international level and still has some deficiencies at Uruguayan level.

Indicator 8 Beneficiary satisfaction levels as measured through perception surveys

Value 0% 70% 80%

(80% individuals; 80% entities)

94% (92% individuals; 95.5%

entities)

Date 12/13/2006 12/15/2012 06/14/2014 06/12/2015

Comments

Surpassed (117% achieved). This indicator was revised during the 2012 restructuring. Data collection has been disaggregated to capture possibly different satisfaction levels among different types of beneficiaries, that is, individuals (mainly scholarships recipients under Subcomponents 2.1 and 2.2 that generate knowledge and technology) versus entities (mainly firms and institutions supported under Component 3 that generate, adapt, and transfer knowledge and technology). The survey captures, among other things, the satisfaction with the support received by the ANII at the various stages of subprojects

ix

implementation and the suitability of the financing support instrument used to achieve the individual's/firm's innovation objective.

Indicator 9 Number of technological innovations/process improvements transferred to the productive sector under Component 3 (Centers, Networks, Stimulating Technological Demand Projects, Alliances) (cumulative)

Value 0 5 7 8

Date 10/23/2012 06/15/2014 06/12/2015 06/12/2015

Comments

Surpassed (114% achieved). This indicator was introduced during the 2012 restructuring, to allow evaluating progress toward achieving one of the Project’s core objectives, namely ‘Strengthen capacity to transfer and adapt knowledge and technology’, and the efficacy of instruments under Component 3, which remained uncovered after the changes to the results matrix introduced in 2010. Given the renewed demand for Technology Centers, the indicator was designed so as to include this type of instrument (see Comments to Indicator 6). The target was revised upward in the 2014 restructuring to reflect the scale-up of activities and extended implementation period. Examples of innovations/process improvements transferred to firms include a system of molecular diagnosis of viral epidemics such as dengue and H1N1; a new technology for the preparation of veterinary vaccines; development of a ‘composed flour’ that recycles non-wheat grains not suitable for export (as soy and oak); quality control of national production of milk; and a new technology for the quality control of biopharmaceuticals.

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Component 1. Support STI Policies and Institutional Strengthening

Indicator 1 Average time to process grant proposals between closing of the call for proposals and final selection (months)

Value 5.7 (N/A for original indicator)

4.0 (4.0 for the original indicator) N/A 4.4

Date 2011 12/15/2012 – 06/12/2015

Comments

76% achieved. Indicator was reformulated in the 2014 restructuring. The original design (‘Average time to process grant proposals between call for proposals and final selection’) created a perverse incentive to shorten the call period, which would have harmed beneficiaries. On the other hand, the new design better captures the efficiency of the implementing agency. According to the new indicator, the reduction in time to process grant proposals with respect to baseline was 1.3 months (76% of the 1.7 months originally foreseen). The average time to process grant proposals would be reduced to 3.5 months if one excludes Sectoral Funds. The processing time is longer in this case because the selection of Sectoral Funds to be financed involves an external committee composed of research institutes and firms (Comité de Agencia). Although the Comité brings a high added

x

value in terms of aligning research with the productive strategy, it represents a further step in the selection process.

Indicator 2 Number of evaluation studies performed or contracted by the National Observatory (cumulative)

Value – 12 14 (2010) 20 (2014)

28

Date 12/13/2006 12/15/2012 06/12/2015 06/12/2015

Comments

Surpassed (140% achieved). As envisaged in the PAD, the Observatory was initially incubated in the ANII as a Monitoring and Evaluation (M&E) unit. However, due to a lack of political consensus, it was never promoted to be an independent institution. The M&E unit of the ANII has performed or contracted 28 studies since its creation, of which 5 are impact evaluations. The indicator was revised in the 2010 restructuring to allow for contracting studies outside the agency. The target was revised upward in the 2010 restructuring to reflect the new design of the indicator and again in 2014 restructuring to reflect implementation to date.

Indicator 3 Number of monitoring indicators tracked annually based on reliable methodologies (annual)

Value – 25 33 (2012) 35 (2014)

35

Date 12/13/2006 12/15/2012 06/12/2015 06/12/2015

Comments

100% achieved. Target was revised upward in the 2012 and 2014 restructurings to reflect progress to date and extended implementation period. Monitored indicators include STI input indicators related to human resources and financing, output indicators on scientific production, indicators of innovation in the private sector, and indicators on the STI culture.

Indicator 4 Number of articles downloaded from Trama Interinstitucional y Multidisciplinaria de Bibliografía On-line (TIMBO) portal (annual)

Value 146,961 (2009) 184,134 (25%

increase)

249,834 (70% increase)

262,326 (78% increase)

316,078 (215% increase)

Date 06/25/2010 12/15/2012 06/12/2015 06/12/2015

Comments

Surpassed (120% achieved). This indicator was introduced in the 2010 restructuring to reflect the decision of financing free access for all Uruguayan citizens to the TIMBO portal (a worldwide database of scientific journals). The target was revised upward in the 2012 and 2014 restructurings, reflecting implementation progress to date and the extended implementation period.

Component 2. Build Human Capital and High-quality Research Teams

Indicator 5 Number of research projects supported (cumulative)

Value 0 14 84 (2012)

134 (2014) 142

xi

Date 12/13/2006 12/15/2012 06/12/2015 06/12/2015

Comments

Surpassed (106% achieved). The target was revised upward in the 2012 and 2014 restructurings to reflect progress to date and the extended implementation period. Eighteen of these subprojects focus on research of High Social Impact (research whose result has a potential to improve living standards of the population) on topics such as road safety, healthy pregnancy, drug addiction, and economic development in small towns. The remaining 124 projects were supported under the umbrella of Sectoral Funds, which prioritize research in strategic sectors (such as Energy, Agro, Fishery, Digital TV) identified by the National Strategic Plan in the STI.

Indicator 6 Annual SCI publications per researcher in project-supported research teams (annual)

Value 0.70 0.77 (10% increase) 0.93 (33% increase)

Date 06/25/2010 12/15/2012 06/12/2015

Comments Surpassed (121% achieved). Baseline for this indicator was introduced in the 2010 restructuring and refers to the average annual SCI publications per researcher, among registered researchers in Uruguay (before receiving support).

Indicator 7 Number of international coauthored articles per researcher in project-supported research teams (annual)

Value 0.48 0.58 (20% increase) 0.60 (25% increase)

Date 06/25/2010 12/15/2012 06/12/2015

Comments Surpassed (103% achieved). Baseline for this indicator was introduced in the 2010 restructuring and refers to the number of international coauthored articles per researcher, among registered researchers in Uruguay (before receiving support).

Indicator 8 Number of graduate programs created with support from the project (cumulative)

Value – 6 5

Date 05/01/2007 12/15/2012 06/12/2015

Comments 83% achieved. Examples include master’s in strategic STI areas such as Management of Technology Firms (ORT University), Engineering (ORT University), Family and Community Medicine (Montevideo University), and Alimentary Technology (UdelaR).

Indicator 9 Number of graduate programs strengthened with support from the project (cumulative)

Value 10 15 21 14 (+13 Technical Tertiary education

programs)

Date 06/25/2010 12/15/2012 06/12/2015 06/12/2015

Comments

67% achieved. Indicator introduced during the 2010 restructuring to reflect Project activities as in the Project Appraisal Document (Subcomponent 2.2) that comprise not only the creation but also the strengthening of graduate programs. If one would consider all tertiary programs strengthened, including technical tertiary programs, the indicator would be 128% achieved. Examples of strengthened graduate programs include Master’s in Chemical Engineering, Environmental Science, Bioinformatics, Mathematical Engineering, Coastal

xii

Management in the Southern Cone, Clinical Pharmacology and PhD programs such as Electrical Engineering, Sciences of Agriculture and Livestock, and Biological Chemistry.

Indicator 10 Number of scholarships awarded to young researchers by the project (cumulative)

Value – 175 660(2012)

1,000 (2014) 1,138

Date 12/13/2006 12/15/2012 06/12/2015 06/12/2015

Comments Surpassed (114% achieved). The target was revised upward in the 2012 and 2014 restructurings to reflect scale-up of activities and the extended implementation period.

Indicator 11 Completion rate of project-supported students in the program allotted timeframe (annual %)

Value N/A 80% 78.1%

Date 12/13/2006 12/15/2012 06/12/2015

Comments 97% achieved

Indicator 12 Number of students completing technical tertiary training supported by the program (cumulative)

Value – 400 200 200

Date 12/13/2006 12/15/2012 06/12/2015

Comments 100% achieved. The target was revised downward during the 2010 restructuring. The 2010 restructuring paper does not inform on the underlying justification.

Component 3. Promote Innovation and Technology Transfer to the Productive Sector

Indicator 13 Number of companies served by supported technology centers or technology transfer operations (cumulative)

Value – 100 216

Date 12/13/2006 12/15/2012 06/12/2015

Comments

Dropped. Indicator dropped in the 2010 restructuring to reflect the shift in focus from Centers and Research Consortia to Alliances and Networks. Indicators related to Technology Centers and Research Consortia were dropped. The value shown at target year refers to the number of companies served by Technology Centers, Networks, and Alliances as of the Project closing date.

Indicator 14 Number of public-private research projects supported (cumulative)

Value – 8 21

Date 12/13/2006 12/15/2012 06/12/2015

Comments

Dropped. This indicator, which referred to Research Consortia, was dropped in the 2010 restructuring to reflect a change in the Project strategy. It was replaced by Intermediate Indicator 19, which explicitly links the concept of public-private research projects to the instrument of Alliances.

xiii

Indicator 15 Private sector cash contributions to supported public-private research partnerships (cumulative)

Value – US$1,200,000 (US$3,000,000

including in-kind)

US$500,000 (cash only)

US$712,883 (cash only)

Date 12/13/2006 12/15/2012 06/15/2014 06/12/2015

Comments

Surpassed (142% achieved). The indicator was revised during the 2010 restructuring to account for cash contribution only, to reflect the design of the contract that the ANII was signing with participating firms. This agreement requires firms to provide a cash contribution equal to at least 30% of the total investment in the Alliance. No requirement for in-kind contribution is specified. The target was revised downward in 2012 to reflect results observed to date.

Indicator 16 Number of informal links (without written contract/agreement) created to support Sector Technology Networks (cumulative)

Value 11 20 36 36

Date 06/25/2010 12/15/2012 06/12/2015 06/12/2015

Comments

100% achieved. This indicator was introduced in the 2010 restructuring to reflect a change in the Project strategy, which implied the introduction of Alliances and Sector Technology Networks. Intermediate Indicators 16, 17, and 18 were designed to capture the progress toward the establishment of Networks: first the generation of informal links between private and public stakeholders, then the formalization of these links by written agreement, and finally the operationalization of Networks. The target was revised upward in the 2014 restructuring to reflect implementation experience to date and the extension of the closing date.

Indicator 17 Number of formal links (with written contract/agreement) created to support Sector Technology Networks (cumulative)

Value 9 12 22 21

Date 06/25/2010 12/15/2012 06/12/2015 06/12/2015

Comments

95% achieved. This indicator was introduced in the 2010 restructuring to reflect a change in the Project strategy, which implied the introduction of Alliances and Sector Technology Networks as financing instruments. The target was revised upward in the 2014 restructuring to reflect scale-up of activities and the extension of the closing date.

Indicator 18 Number of operational Sector Technology Networks established (cumulative)

Value – 3 5 5

Date 06/25/2010 12/15/2012 06/12/2015 06/12/2015

xiv

Comments

100% achieved. This indicator was introduced in the 2010 restructuring to reflect a change in the Project strategy, which implied the introduction of Alliances and Sector Technology Networks. The operational Networks address strategic areas such as improving the quality of exports in the primary sector, boosting tourism, and providing a business incubator for innovative small and medium enterprises (SMEs). The target was revised upward in the 2012 and 2014 restructurings to reflect scale-up of activities and the extended implementation period. The definition of the indicator was slightly modified during the 2014 restructuring to make it more stringent. The new definition considers only ‘operational’ Networks, which condition is satisfied once the contract establishing the Network has been signed and the first disbursement has been completed.

Indicator 19 Number of operational Alliances supported (cumulative)

Value 3 8 17 21

Date 06/25/2010 12/15/2012 06/12/2015 06/12/2015

Comments

Surpassed (123% achieved). This indicator was introduced in the 2010 restructuring to reflect a change in the Project strategy, which implied the introduction of Alliances and Sector Technology Networks as financing instruments. Alliances are public-private partnerships created with the aim of solving a specific innovation problem and are dissolved after the achievement of the objective. The target was revised upward in the 2012 and 2014 restructurings to reflect scale-up of activities and the extended implementation period. The definition of the indicator was slightly modified during the 2014 restructuring to make it more stringent. The new definition considers only ‘operational’ Alliances, which condition is satisfied once the contract establishing the Alliance has been signed and the first disbursement has been completed. Examples of objectives of operational Alliances include improvement of productivity and sustainability of soy cultivation; development of new technologies for quality control of bio-treatments; and development of a system of molecular diagnosis of viral epidemics such as dengue and H1N1.

Indicator 20 Number of entities/companies/institutions participating in the Alliances supported (cumulative)

Value 4 10 57 63

Date 06/25/2010 12/15/2012 06/12/2015 06/12/2015

Comments

Surpassed (110% achieved). This indicator was introduced in the 2010 restructuring to reflect a change in the Project strategy, which implied the introduction of Alliances and Sector Technology Networks as financing instruments. The target was revised upward in the 2012 and 2014 restructurings to reflect scale-up of activities and the extended implementation period. If one stakeholder participates in more than one Alliance, it is counted only once.

Indicator 21 Number of technological demand support projects (cumulative)

Value – 6 3 3

Date 10/23/2012 06/14/2014 06/12/2015 06/12/2015

xv

Comments

100% achieved. This indicator was introduced in the 2012 restructuring to reflect the increase in scope of Component 3 to include Stimulating Technological Demand projects as a new financing instrument. The target was revised downward in the 2014 restructuring since this type of activity was eventually closed down.

Indicator 22 Number of operational Sector Technology Centers (cumulative)

Value – 2 3 5

Date 10/23/2012 06/14/2014 06/12/2015 06/12/2015

Comments

Surpassed (167% achieved). This indicator was introduced in the 2012 restructuring to reflect the increased demand for Sector Technology Centers and, consequently, the ANII strategic decision to promote this type of instrument. The target was revised upward in the 2014 restructuring to reflect scale-up of activities and the extended implementation period. The definition of the indicator was slightly modified during the 2014 restructuring to make it more stringent. The new definition considers only ‘operational’ Centers. The formal legal and operational requirements to meet this condition were specified in details in the 2014 Restructuring Paper. Examples of operational Centers are the Center of Innovation in Engineering, the Center of Information and Communication Technology, Unit of Innovation in Alimentary Technology, and the Center of Industrial Extensionism. Besides the 5 operational Centers, 3 more are currently being discussed and approaching the contract signature stage.

G. Ratings of Project Performance in ISRs

No. Date ISR Archived

DO IP Actual Disbursements

(US$, millions) 1 05/17/2007 Satisfactory Satisfactory 0.00 2 12/18/2007 Satisfactory Satisfactory 0.44 3 04/18/2008 Satisfactory Moderately Satisfactory 0.44 4 11/21/2008 Satisfactory Moderately Satisfactory 0.44 5 05/30/2009 Satisfactory Satisfactory 1.02 6 11/20/2009 Satisfactory Satisfactory 1.51 7 05/02/2010 Satisfactory Satisfactory 4.49 8 06/18/2010 Satisfactory Satisfactory 5.87 9 01/24/2011 Satisfactory Satisfactory 7.94 10 08/10/2011 Satisfactory Satisfactory 11.86 11 03/03/2012 Satisfactory Moderately Satisfactory 11.86 12 10/02/2012 Moderately Satisfactory Moderately Satisfactory 17.63 13 05/27/2013 Moderately Satisfactory Satisfactory 18.75 14 11/23/2013 Moderately Satisfactory Satisfactory 21.15 15 05/26/2014 Satisfactory Satisfactory 22.14 16 11/26/2014 Moderately Satisfactory Satisfactory 24.11

xvi

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in US$, millions

Reason for Restructuring & Key Changes Made

DO IP

01/22/2008 S S 0.44

Restructuring driven by amendment to legal documentation: - Inclusion of default interest and the front-end fee amount in the Loan Agreement

12/01/2008 S S 0.44

Restructuring driven by amendment to legal documentation: - Designation of the ANII as the implementing agency

06/25/2010 S S 6.0

Restructuring driven by changes in the implementation strategy: - Amend Project documentation to finance Uruguay’s access to the international database of scientific publications (TIMBO). - Reallocate proceeds. - Extensively revise the Results Framework to reflect the shift of focus from promoting Centers and Consortia to smaller instruments such as Alliances and Networks under Component 3.

12/12/2012 MS MS 17.7

Restructuring driven by changes in the implementation strategy: - Extend closing date (from 2012 to 2014), mainly to allow implementation of Centers under Component 3. - Reallocate proceeds, mainly to respond to the higher demand for scholarships under Component 2. - Extensively revise the Results Framework.

08/14/2014 S S 22.4

Restructuring driven by the change in the closing date: - Extend closing date (from 2014 to 2015). - Minor changes to the Results Framework.

xvii

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design 1. The Promoting Innovation to Enhance Competitiveness Project was a response to the Government's goal of promoting scientific and technological progress (Uruguay Innovador pillar in the Government strategy). The Project included a comprehensive set of mutually reinforcing activities. Component 1 focused on supporting science, technology, and innovation (STI) policies, including the development of the National Strategic Plan for STI, and the creation and institutional strengthening of the National Research and Innovation Agency (Agencia Nacional de Investigación e Innovación, ANII). Component 2 offered direct financing for research subprojects and scholarships for graduate students and young researchers and strengthened the offer of graduate and technical tertiary programs. Component 3 financed public-private innovation partnerships. If successful, the Project would strengthen Uruguay's capacity to generate, transfer, and adapt knowledge and technology.

1.1 Context at Appraisal 2. Uruguay had experienced an impressive recovery and rapid growth since the crisis and deep recession of 1998–2003. Real gross domestic product had been growing at an average annual rate of 5.2 percent between 2004 and 2006.1 The challenge was to sustain robust gross domestic product growth and break with the slow-growth pattern of previous decades, by coupling the recently achieved macroeconomic stability with increased productivity. Understanding that this would have been possible only by finding new and better ways of doing business, the Government of Uruguay gave high priority to innovation during 2005–2009 by including it as one of the six pillars of the National Development Program.2

3. At the time of appraisal, total investment in STI in the country had been modest and unstable. STI policies and programs had evolved in an ad hoc manner, with the lack of coordination resulting in the duplication of efforts in some areas, while other problems had remained unattended. The country had yet to develop data collection structures and validation processes that ensure a continuous flow of reliable STI statistics, and there was a lack of information on the effectiveness of public STI expenditure. Uruguay lacked an adequate supply of advanced human capital, especially in terms of graduate and technical tertiary education, and a critical mass in research. The research agenda was not focused on the problems of the productive sector but was basically defined by researchers. Investment in research and development (R&D) had been mainly public and concentrated in the University of the Republic. There was insufficient capacity to transfer and adapt technology in private companies, especially because of underdeveloped links between demand and supply of knowledge and technology.

4. In 2001, there was an attempt to institutionalize innovation policy with the establishment of the National Directorate of Science, Technology, and Innovation (Dirección Nacional de Ciencia, Tecnología e Innovación, DINACYT) as a department within the Ministry of Education and Culture

1 World Development Indicators (WDI). 2 The 2005–2009 Government’s program ‘El Gobierno de Cambio - La Transición Responsible’ introduced a new national development strategy for the country articulated along six key complementary themes: enhancing productivity (Uruguay Productivo), fostering social development (Uruguay Social), promoting innovation (Uruguay Innovador), strengthening democracy (Uruguay Democrático), enhancing regional and global integration (Uruguay Integrado), and promoting policies which consolidate and promote Uruguayan culture (Uruguay Cultural).

2

(MEC). This institutional framework had not proven effective. The MEC had been moderately successful in supporting basic science but little progress had been made with regard to stimulating public-private research linkages and innovation. DINACYT’s funding had been volatile, and the implementation of innovation programs remained dispersed among other agencies.

5. As of 2006, Uruguay relied on the Inter-American Development Bank (IDB) for external financial and technical assistance on STI, through the two consecutive projects ‘Science and Technology Program’ (1991–2000) and ‘Technological Development Program II’ (2001–2007). To deal with the ambitious challenge of both institutionalizing innovation policy and expanding the set of instruments to support STI, the Government looked for support from more international organizations. The World Bank was recognized as a preferred partner, given its long-standing international experience on STI and its encouragement of local dialogue on innovation.

6. The Promoting Innovation to Enhance Competitiveness Project was a response to the Government's goal of promoting scientific and technological progress to jump-start growth (Uruguay Innovador pillar in the Government strategy). It was also fully aligned with the FY05/10 Country Assistance Strategy II pillar to promote sustainable growth by helping increase total factor productivity, improving access to highly skilled individuals, enhancing the competitiveness of existing sectors, and facilitating the growth of emerging sectors (Report No. 31804-UY). In addition, the Project indirectly aimed at supporting the creation of new and more productive jobs helping to enhance living standards, which corresponded to pillar III of the FY05/10 Country Assistance Strategy. Finally, the Project built on the analytical work ‘Uruguay: Sources of Growth, Policies for the Development of Human Capital, Integration, Competition and Innovation’, June 14, 2005 (Report No. 31737-UY).

1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 7. The PDO is to strengthen Uruguay's capacity to generate, transfer and adapt knowledge and technology by supporting (i) the institutional framework for STI; (ii) investments in human capital and high-quality research teams; (iii) technology transfer and private sector innovation; and (iv) cross-sectoral and international research collaboration and mobility of researchers between public institutions and the productive sector.3

8. These objectives were translated into the following key indicators and targets:

40 percent of annual public R&D funding administered by the ANII (annual) 50 research groups with more than 3 researchers registered in the national researcher

classification system (cumulative) 50 internationally published Science Citation Index (SCI) scientific articles (cumulative) 240 graduates from domestic master’s and PhD programs (annual) 40 project-supported researchers inserted into industry (cumulative) 40 percent of variable costs of supported technology centers generated by the sale of services

(annual) 4 patent applications submitted by supported research consortia (cumulative) 70 percent of beneficiaries were satisfied, as measured through perception surveys

3 The statement of the PDO in the Loan Agreement (pag. 5) and in the PAD (pag. 11) are identical.

3

1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 9. The PDO was not revised. Key indicators and targets were extensively revised during the 2010 and 2012 restructurings and to a minor extent during the 2014 restructuring. The following are the revised/added key indicators (more details on the changes applied and relative justification can be found in the Data Sheet):

1,000 internationally published scientific articles (annual) - adaptive/opportunistic restructuring

600 graduates from domestic master’s and PhD programs (annual)- opportunistic restructuring

30 project-supported researchers inserted into industry (cumulative)- adaptive/opportunistic restructuring

80 percent of beneficiaries were satisfied, as measured through perception surveys- opportunistic restructuring

7 technological innovations/process improvements transferred to the productive sector under Component 3 (Centers, Networks, Stimulating Technological Demand Projects, and Alliances) (cumulative) - adaptive restructuring

1.4 Main Beneficiaries 10. The direct target group were national stakeholders directly involved in STI activities: the ANII (especially through activities of institutional strengthening under Component 1); research institutes and researchers (mainly by supporting research subprojects and investigators, as well as graduate and technical education under Component 2; and through links and activities generated with the private sector under Component 3); and the private sector in Uruguay (primarily through activities under Component 3). The indirect target group could possibly be identified with the Uruguay population as a whole if positive spillovers with regard to increased productivity, sustainable growth, and enhanced living standards from innovation materialize.

1.5 Original Components (as approved) 11. Component 1 aimed at supporting STI policies and institutional strengthening. It was organized into three subcomponents: (1.1) Support the development of public STI policies; (1.2) Support the ANII; and (1.3) Set up and support the STI Observatory. The first subcomponent is focused primarily on offering support to draft the National Strategic Plan for STI (PENCTI), including consultation and dissemination activities on STI policies. The goal of activities under the second subcomponent was the creation and strengthening of the ANII as the coordinator and implementing agency of national STI policies. Finally, the third subcomponent focused on ensuring the setup of a strong monitoring and evaluation (M&E) system for STI policies, initially to be incubated within the ANII and subsequently to be designed as an independent institution (observatory). Overall, activities under this component were strategic to set up the institutional framework to achieve the PDO.

12. Component 2 intended to build human capital and high-quality research teams. This component was also organized into three subcomponents: (2.1) Promote excellence in science and

4

technology research in priority areas; (2.2) Increase the stock of advanced human capital; and (2.3) Strengthen priority technical tertiary education programs. Activities under the first subcomponent provided direct funding to research projects in strategic areas identified by the PENCTI, such as exports, information and communication technology (ICT) and biotechnology, as well as research of high social impact. A multistakeholder, multidisciplinary approach was to be preferred. Subcomponent 2.2 focused on the ‘production’ of young researchers. It included a portfolio of scholarships for undergraduate and graduate students, as well as the development and strengthening of domestic master’s and doctoral programs. Finally, Subcomponent 2.3 targeted ‘production’ of technicians, by developing and strengthening technical tertiary education programs.

13. Component 3 focused on promoting innovation and technology transfer to the productive sector. It included two subcomponents which, in their original design, reflected the two instruments to be sponsored. Subcomponent 3.1 covered activities to develop Sector Technology Centers, physical spaces gathering STI specialists able to offer/sell technology services to firms in strategic sectors. Subcomponent 3.2 supported public-private research consortia, aimed at facilitating collaboration between demand (private sector) and supply (research institutes and academia) of knowledge and technology.

1.6 Revised Components 14. The 2010, 2012, and 2014 restructurings allowed for an expanded set of activities under Component 1 (inclusion of the Trama Interinstitucional y Multidisciplinaria de Bibliografía On-line [TIMBO] portal to coordinate and expand access to scientific publications) and Component 3 (to comprise Sector Technology Centers, Sector Technology Networks, Alliances, and Stimulating Technological Demand Projects).

1.7 Other Significant Changes 15. In 2008, as approved by the first Project restructuring, the Loan Agreement was amended to eliminate interest waivers and include default interest and the front-end fee amount. The disbursement table was changed accordingly. Fund allocation was further revised in 2010, with the inclusion of a new category ‘Access to Scientific Publications’, to finance the TIMBO portal, and in 2012, when resources from the Project Preparation Facility (PPF) that were not used during project preparation were reallocated to Component 2 to finance additional scholarships. Other significant changes include the formal designation of the ANII as the implementing agency (second project restructuring, 2008) (more details can be found in section 2.2).

2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 16. The Bank had strongly positioned itself as a supporter of innovation policy in Uruguay, building on the analytical work Uruguay: Sources of Growth, Policies for the Development of Human Capital, Integration, Competition and Innovation (2005, Report No. 31737-UY). First, the study stressed the need to build an integrated system of innovation policy to improve coordination between STI programs. This would have required a deeper institutionalization of STI policy and the integration of STI programs under the umbrella of one single implementing agency able to ensure

5

coordination and budget sustainability. Second, the report highlighted the importance of creating linkages between firms and R&D activities by fostering public-private partnerships for innovation and developing a diversified set of sectoral innovation clusters. These policy recommendations were incorporated in the Project design.

17. The Project design also benefitted from operational experiences from the Bank-financed projects in Mexico (P044531), Venezuela (P066749), Chile (P055481), Brazil (P038947), India (P009895), and Korea (I P004102, II P004117, and III P004134). Experiences in Venezuela and Chile suggested the value of strengthening domestic graduate education to reduce ‘brain drain’. The Mexico and Brazil projects proved the effectiveness of a combined approach involving both STI-supply-related (building human capital) and demand-related (promoting innovation in the private sector) instruments. The Indian and Korean experiences confirmed that the Bank involvement would have a positive impact in facilitating the creation of innovative partnerships between firms and research institutions and that both a sound STI policy framework and a strong implementing agency were crucial for the success and sustainability of this type of activity. All these lessons were highly relevant for the case of Uruguay and were incorporated in the Project design. Finally, Project preparation drew on the experience of the two consecutive STI projects financed by the IDB in Uruguay to identify policy areas yet to be developed.

18. Coordination among multilateral agencies was ensured to avoid overlaps and deliver the maximum value added from collaboration. Table 1 sums up the coordination of external support by area of work, based on the specific technical experience and resource commitment of each multilateral agency.

Table 1. Support from International Partners to Uruguay STI Policy

Organization Specific Support World Bank (Promoting Innovation to Enhance Competitiveness, 2007–2015; US$26 million + US$6.5 million (Government co-financing)

Institutional strengthening. Support in drafting PENCTI; Strengthening of ANII; M&E system

Human capital. Multi-disciplinary research projects: Scholarships for domestic undergraduate and graduate students; Strengthening domestic graduate and technical tertiary programs.

Productive sector. Public-private partnerships for innovation through Alliances, Networks, and Centers.

IDB (Technological Development Program II, 2008–2014; US$34 million)

Human capital. International mobility of graduate students. Productive sector. Development of specific technology/innovation

within firms; international cooperation among firms; financing the hiring of qualified human resources by firms.

European Union (Uruguay Innova, 2008–2010; US$10.4 million)

Strengthening institutes of excellence in R&D such as Pando Park and the Pasteur Institute.

19. Project objectives were important for Uruguay, reflecting the strong focus of the Government on promoting scientific and technological progress. The Project's innovative, comprehensive scope was justified by the lessons learned from operational experience and analytical studies, which pointed at the need to simultaneously build the main pillars of an innovation system, namely strengthening the STI institutional framework, building human capital, and promoting an innovation culture within the productive sector. The choice of investing in admittedly complex instruments, such as public-private partnerships for innovation under Component 3, was both robustly, theoretically founded and

6

innovative in the region. Overall, the Project design was commendable in terms of fostering innovative policy instruments. The preparation process was highly participative. 20. Given the complexity of innovation policy as an area of intervention, clarity in the design could have been improved by better specifying the causal links between the Project's activities, key indicators, and the PDO, as well as the definition of the PDO itself (especially the concepts of generating, transferring, and adapting knowledge and technology). The assessment of risk did not explicitly take into account the lack of associative culture between the private and the public sector in Uruguay, which contributed to slow implementation of activities under Component 3. The assessment in the Project Appraisal Document (PAD) only takes into account "potentially less demand than expected from the private sector for technology services" of centers and consortia (PAD, pag. 21), but not for the difficulty in setting up the centers and the consortia themselves. Although lessons learned from operational experiences all suggested that "there is a fairly long 'learning' period for all participants in such projects [...] it may be typical to see slow starts (reflected in slow disbursements) in the beginning" (India Industrial Technology Project ICR, Report 18105), no specific action was designed to mitigate this risk, which indeed materialized and required implementation adjustments.

2.2 Implementation 21. Although Project implementation took 2.5 years longer than planned, it is still evaluated as satisfactory, because the extension of the closing date resulted in the scale-up of activities. The Project was restructured five times (Level II restructuring) to continuously adapt to changing circumstances. The PDO was never changed, but the way of achieving it was adapted. The first restructuring (2008) allowed the inclusion of default interest and the front-end fee amount in the Loan Agreement. The second restructuring (2008) officially designated the newly created ANII as the implementing agency. The third restructuring (2010) allowed for financing a new activity under Component 1 (TIMBO portal) and revised the results matrix so as to reflect the shift of focus from promoting Centers and Consortia to smaller instruments such as Innovation Alliances and Networks under Component 3. The fourth restructuring (2012) responded to the higher-than-expected demand for scholarships under Component 2, as well as an increasing demand for activities under Component 3, particularly Centers. The closing date, funds allocation, and results matrix were changed so as to assist implementation given these new conditions. The fifth restructuring (2014) approved a further extension of the closing date to further scale up activities under Components 2 and 3. Targets and indicators were changed accordingly. All Project restructurings were effective in meeting the proposed PDO and supported a smoother implementation.

22. Component 1. The elaboration of the PENCTI4, directly supported by the Project's activities, provided a strategic framework for innovation policy. It clearly identified the priority areas to concentrate financing efforts to promote STI, therefore contributing to better define the ANII's institutional mandate and areas of intervention. The risk of lack of collaboration between Government

4 Approved by Ministerial Decree in 2010.

7

ministries, identified as substantial in the PAD (pag. 20), was partly mitigated by the Inter-Ministerial Cabinet for Innovation (Gabinete Ministerial de la Innovación, GMI).5

23. The ANII was officially designated as the Project implementing agency in the second Project restructuring (2008). Consultancy activities, workshops, and studies financed through the PPF and Component 1 were effective in mitigating the risk of delays (modest, according to the PAD) in the agency becoming fully operational and ensuring smooth financial and procurement management. A proper design and quick operationalization were, in turn, crucial for the satisfactory implementation of the Project as a whole.

24. Initially, the Project funds allocation included the financing of the ANII's personnel, mainly external consultants, under the category of operating costs. Subsequently, the Government opted for a more stable organization of human resources to reduce turnover and ensure continuity of operations, and the ANII opened staff positions to be financed with national budget resources. This choice was rewarded by a reduction in staff turnover from 13 percent in 2007 to 5 percent in 2014.

25. In 2008, the ANII identified a strategic opportunity to strengthen Uruguay’s capacity to generate, transfer, and adapt knowledge and technology. Through the TIMBO portal, Uruguay aimed at being the first country in the world to offer access to the latest scientific publications to the whole population in the country. The Bank team prepared a Project restructuring (2010) to reallocate the funds initially designated to the ANII's personnel under operating costs, and create a new category ‘Access to Scientific Publications’ to finance the TIMBO portal.

26. Activities under Subcomponent 1.3 allowed for implementing the creation of a strong M&E unit to be incubated in the ANII (see annex 2 for more details). However, the lack of political consensus around the question of what type of STI Observatory would be needed, impeded the setup of an external, independent institution with M&E functions, envisaged in the PAD as a successive step in implementation of Subcomponent 1.3.

27. Component 2. The lines of activity under Subcomponent 2.1 were grouped under two types of instruments to avoid overlaps and multiplicity of application procedures: High Social Impact subprojects6 and Sectoral Funds.7,8 With respect to the original design, scholarships offered under Subcomponent 2.2 were limited to (a) support to undergraduate students willing to work in high-quality research environments (Becas de Iniciación a la Investigación); (b) scholarships for master’s and doctoral students in domestic programs (Posgrados Nacionales).9

5 The GMI represents the strategic political level of the national innovation system, being responsible for formulating innovation policy guidelines at the country level. 6 High Social Impact subprojects, already envisaged in the PAD, constitute an instrument to financially support research aimed at improving living conditions. 7 Sectoral Funds provided financing for multidisciplinary research groups willing to develop knowledge and innovation in areas of research which were underdeveloped or of strategic importance to Uruguay. Key priority areas were identified based on the PENCTI. 8 The scope of this last activity was increased as a result of the extension of the closing date approved in the 2014 restructuring, allowing for financing an additional Sectoral Fund in Energy. 9 Demand for scholarships was higher than expected, especially for the Iniciación type. To accommodate this, the scope of the instrument was increased in the 2012 restructuring, which allowed for extension of the closing date and for the reallocation of resources from PPF funds that were not used during preparation to finance additional scholarships. The

8

28. Initially, the Project also included the award of partial stipends for the introduction of young researchers in the productive sector. However, the one call that was opened for this type of scholarship received only applications from firms that were evaluated as not qualified to offer proper training for young researchers. The instrument was hence closed down with no beneficiaries. The strategy to introduce researchers into the productive sector shifted to a combined use of Sectoral Funds and High Social Impact subprojects on the one hand (directing researchers' work toward strategic areas of interest for the productive sector) and Alliances, Networks, and Centers under Component 3 (which actively promoted constructive contact between researchers and firms).10

29. Component 3. This was the most challenging component, given that it focused on a specific type of activity—public-private partnerships for STI—that was highly innovative in the country and in the region and presented higher implementation risk. Because of the lack of collaborative culture between the public and private sector in Uruguay, especially in the field of innovation, the demand-driven approach did not work well. As a result, disbursement under this component began only in 2010 and was slow until 2012, originally the Project closing year.

30. Several initiatives were taken to improve implementation of this component. The set of activities was expanded to include Alliances, Networks, and Stimulating Demand Projects so as to offer different and more flexible options of collaboration between public and private stakeholders.11 The ANII adopted a proactive approach to promote these instruments among research institutes, universities, and firms, pursuing potential participants and providing advice and hands-on assistance to candidates. In particular, a special unit was created for intensifying the relationship with the productive sector, and the agency started collaborating with Chambers of Commerce and other local actors. The Bank also supported specific knowledge exchange activities. Thanks to intense promotion work among firms, the demand for Alliances and Networks grew rapidly.

31. As of 2012, several changes allowed for more favorable conditions to invest in Centers. First and most important, two years of implementation of Alliances and Networks provided successful examples of public-private innovation partnerships among local actors. Second, a change in ANII’s management coincided with a renewed interest for Centers to further boost innovation in the productive sector. Third, although the Bank team had continuously supported this type of instruments by sharing knowledge and experience, in 2012 a special effort was made by organizing a study tour in Spain and Denmark, to allow the Government of Uruguay to directly learn from these pioneers in the creation of Technology Centers. In 2013, Centers were officially launched in Uruguay and

scope of Posgrados scholarships was also increased, due to the extension of closing date approved in the 2014 restructuring. 10 Activities under Subcomponents 2.2 and 2.3 also contributed to the creation and strengthening of graduate and technical tertiary programs in key strategic STI areas identified by the PENCTI. Implementation of these activities was in line with the PAD design and did not encounter any implementation issue. 11 Alliances and Networks required less financial and organizational commitment than Centers and were more suitable for less mature firms. As such, they could serve as successive steps to build confidence around the idea of public-private partnerships for innovation and set the basis for the creation of Centers in the future. Alliances, in particular, represented flexible research consortia; they were created to find a solution to a specific innovation demand, and dissolved thereafter. Networks, instead, implied more long-term relationship. Neither case, however, required the financial and organizational structure to sell innovation and technology services that was instead demanded in the case of Centers.

9

extremely interesting examples have emerged since then.12 Overall, the rapid catch-up observed in terms of implementation of all instruments under Component 3 suggests that the Project built the base for more of these activities in the future.

32. As a consequence of implementation decisions, actual disbursement per component differed from estimates at appraisal. Total actual cost under Component 1 was US$9.3 million (+US$6.2 million with respect to PAD estimates), US$18.3 million under Component 2 (+US$1.6 million), and US$4.5 million under Component 3 (−US$7 million).13 Despite the number of financed public-private partnerships under Component 3 that largely surpassed estimations at appraisal, the financial scale of each one was lower than expected (partly due to the requirement for firms to provide a cash contribution equal to at least 30 percent of the total investment in the partnerships), leading to a lower amount of funds disbursed under Component 3. On the other hand, the TIMBO (Component 1) was not envisaged at the time of appraisal. Note that the choice to provide universal access to scientific publications and channel it through a unique public platform has led to a higher number of downloads and a drop in unitary costs (see sections 3.2, 3.3 and annex 3 for more details), and it has been recently recognized as international best practice by the Global Innovation Index (GII 2015).14 Finally, demand for scholarships under Component 2 was higher than expected. To put this into context, it is worth recalling that the ANII is the only provider of scholarships for young researchers to gain experience in research institutes, and by far the major public provider of Posgrado scholarships.15 Overall, the changes introduced during implementation are positively evaluated with regard to supporting the Project efficacy and efficiency.

2.3 Monitoring and Evaluation (M&E) Design, Implementation, and Utilization 33. Design. The M&E system had some weaknesses. In particular, a clearer results chain explaining the causality between activities financed under each component, key indicators, and the PDO could have better guided Project implementation. The original design of the results matrix took into account the international literature, but in some cases it was not appropriate in evaluating the specific country context (for example, Uruguay's position in terms of patents). Some of the described shortcomings could be explained by the fact that the M&E system in the ANII was built practically from scratch together with the Project itself. Additionally, to date there are no clear international standards on which indicators better capture progress in innovation. Finally, although both the PAD and the Results Matrix recommended the completion of evaluation studies, no specific indication was given on the type, scope, and possible techniques to apply.

12 The two successive extensions of the closing date (first from 2012 to 2014 and eventually to 2015) allowed for full implementation and disbursement under Component 3, including the creation of Centers and financing of an increased number of Alliances and Networks. 13 The increase in funding in Components 1 and 2 does not match the decrease in Component 3 because of resources freed from unallocated funds and unused PPF. Note that not all of this shift in funds required changes to the disbursement categories. See annex 1 for more details. 14 For more information, see WIPO's website: http://www.wipo.int/export/sites/www/pressroom/en/documents/gii_2015_infographic2.pdf. 15 Universidad de la República provides a very limited number of scholarships for postgraduate students.

10

34. Implementation. The ANII demonstrated strong capacity and built a reliable M&E unit with qualified staff. The appropriate data was collected on a timely basis and made available to the Bank16. Collection mechanisms were appropriate and attention was paid to ensuring quality. The ANII conducted/contracted 28 studies (technical and sectoral reports, surveys, and qualitative and quantitative evaluations), among which five impact evaluations (two on instruments supported by the Project, (Iniciación a la Investigación and Posgrado scholarships). Data for an additional impact evaluation on Sectoral Funds is being collected at the time of writing the ICR. The M&E arrangements appear to be sustainable beyond the Project lifespan as they were extended to other activities and are now part of the ANII regular work program.

35. Utilization. The ANII was proactive in promoting the utilization of the compiled data on STI. Specific reports were sent to the Government and other relevant institutions for STI policy. The indicators were used for project management. STI studies and a bulletin of indicators are published online every year and events are organized to raise knowledge among the public opinion. Importantly, the ANII has an open data policy that includes free access to STI surveys microdata.

2.4 Safeguard and Fiduciary Compliance 36. Environmental and social safeguards. The Project had an environmental category C. No safeguard was triggered by the Project.

37. Financial Management. The financial management (FM) performance of the Project has been consistently satisfactory and FM requirements were fully complied with throughout the entire implementation period. Moreover, project unqualified audited financial statements were received by the Bank on time, despite some minor delays at project inception, mostly due to the fact that ANII was a recently established entity at that time.

38. Procurement. All procurement was done under the provisions stipulated in the Legal Agreement and procedures in the operational manual. For subprojects carried out by the grantee's research institutions, procurement followed sector commercial practices approved by the agency. The operations manual was improved during Project execution and procedures for subprojects were improved. No serious issues were found during ex post review missions and in general, the performance of the agency was satisfactory.

2.5 Post-Completion Operation/Next Phase 39. Several factors suggest that there are the lega, financial and institutional capacity means of sustaining the reforms and institutional capacity supported by the loan. These are discussed in details in section 4.

3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation

16 The only indicator suffering from delays was the number of graduates from domestic master’s and PhD programs, the collection of which, was the responsibility of the Ministry of Education and was usually published 12 or more months after closing of the school year.

11

40. Relevance of objective. Pre-2010 restructuring rating: High; 2010–2012 rating: High; Post-2012 restructuring rating: High.

41. The objective was strategically aligned to the country’s strategy and the operation remained important to the achieving country. The draft 2015–2019 national development strategy reaffirms the importance of innovation to ensure sustainable growth in Uruguay, and the National System of Competitiveness Law Bill states that "…it is crucial to progress in areas such as human capital, innovation, and technology” (Presidency of Uruguay, March 2015)." The Project objective is also aligned with Objective 6 in the draft 2015–2020 Country Partnership Framework, aimed at increasing R&D opportunities to facilitate integration into global value chains.

42. Relevance of design and implementation. Pre-2010 restructuring rating: Substantial; 2010–2012 rating: Substantial; Post-2012 restructuring rating: Substantial.

43. The Project design was both comprehensive and innovative, allowing for the simultaneous development of the strategic pillars to build an innovation system. The planned activities were consistent with Project objectives. The restructurings allowed the Project to adapt to changing circumstances and introduce balanced modifications to designed activities so as to ensure that the set of instruments remained both adequate and sufficient to reach the PDO.17

3.2 Achievement of Project Development Objectives 44. To assess the Project’s achievement, this ICR considers the following dimensions of the PDO: (a) to strengthen Uruguay's capacity to generate knowledge and technology and (b) to strengthen Uruguay's capacity to transfer and adapt knowledge and technology.18 Clearly, the two concepts are highly interrelated, and the same activities and indicators may be considered when evaluating achievement of the two dimensions. The assessment relies on the Project’s key indicators, intermediate indicators, supplementary indicators collected at the ICR stage, and results from evaluations published by the ANII. Split ratings have been applied to reflect the extensive changes

17 As an example, the 2010 restructuring acknowledged the low demand for Centers and allowed for other types of instruments (Alliances and Networks) under Component 3 to achieve the project objectives. Given the changed circumstances documented in Section 2.2 (Implementation), the 2012 restructuring proactively sought an opportunity to expand the set of instruments to achieve the PDO by including Alliances, Networks, and Centers. 18 The decision to evaluate together the concepts of transfer and adaptation of knowledge and technology relies on the following considerations. First, a review of the sectoral literature (OECD 2005; European Community 2007; Sabater 2009; U.S. Department of Energy 2011) and of the international definitions (Oslo Manual 1997, OECD 2015, UNCTAD, GII) revealed that although the two concepts are different (as specified in footnote 25), they are usually used to refer to the same type of innovation activity but from different angles. There is no internationally established set of transfer and adaptation indicators suitable to be supplied in a separate way. More details from this sectoral literature review can be found in annex 10. Additionally, there are particular considerations related to the project itself. Although the PAD did not specify the definitions of generation, transfer, and adaptation, it usually referred to the last two together. For instance, the country diagnostics at page 6 highlights the "insufficient capacity to transfer and adapt technology in private companies", while the Data Sheet and components description explain that instruments under Component 3 are designed such that they "promote technology transfer and accelerate the adaptation of new technologies and innovations by firms" (PAD 2007). The choice of considering transfer and adaptation as one unique dimension of the PDO has been agreed at the Quality Enhancing Review meeting of this ICR by the attending innovation specialists.

12

to key indicators introduced in the 2010 and 2012 restructurings, while changes introduced in 2014 were not considered to make a material difference to the achievement of PDO.19

Table 2Error! No sequence specified.. Causal Linkages between the PDO, Revised Key Indicators, and the Project's Components (according to Review during the ICR Stage)

Outcomes Outputs Key Indicators and targets Components Strengthen Uruguay’s capacity to generate knowledge and technology

Support the institutional framework for STI

80% of beneficiaries were satisfied 40% of annual public R&D funding

administered by the ANII, annual (dropped in 2012)

Component 1

Support investments in human capital and high-quality research teams

1,000 internationally published scientific articles

600 graduates from domestic master’s and PhD programs

50 research groups with more than 3 researchers registered in the national researcher classification system (dropped in 2010, not measurable )

Component 2

Support technology transfer and private sector innovation

7 technological innovations/ process improvements transferred to the productive sector under Component 3

Component 3

Strengthen Uruguay’s capacity to transfer and adapt knowledge and technology

Support the institutional framework for STI

80% of beneficiaries were satisfied 40% of annual public R&D funding

administered by the ANII, annual (dropped in 2012)

Component 1

Support technology transfer and private sector innovation

4 patent applications submitted by supported research consortia, cumulative (dropped in 2010)

7 technological innovations/ process improvements transferred to the productive sector under Component 3

40% of annual variable costs of supported technology centers generated by the sale of services (dropped in 2010, not measurable)

Component 3

Support cross-sectoral and international research collaboration and mobility of researchers between public institutions and the productive sector

30 project-supported researchers were introduced into industry

Component 2

Component 3

Strengthen Uruguay's Capacity to Generate Knowledge and Technology20 45. Rating. Pre-2010 restructuring rating: Substantial; 2010–2012 rating: Substantial; Post- 2012 restructuring rating: High.

46. The objective was achieved. Efficacy in achieving this dimension of the PDO increased from Substantial to High throughout Project implementation.21 Three out of four key targets effective up

19 As specified in the Data Sheet (section H) and discussed in section 2.2 of this ICR, the 2014 restructuring introduced only minor changes to the results matrix. 20 This is intended as the country's capacity to create brand new knowledge and technology. 21 Although progress against this objective was very successful already from the early stages of the project (as reflected by the intermediate indicators), the progressive change in rating reflects, on one side, the useful implementation

13

to 2010 were largely surpassed. Similarly, three out of four key targets effective up to 2012 were largely surpassed, and all four key targets effective up to 2015 were surpassed. To overcome the discussed limitations of the key indicators in fully capturing achievement of the objective, table 3 presents information on additional indicators. Overall, these data confirm that the Project's activities contributed to increase the level of inputs in terms of (a) institutional capacity building; (b) human capital development; and (c) incentives to promote innovation among the private sector and translate these inputs into generation of knowledge and technology.

Table 3. Results: Strengthening Uruguay's Capacity to Generate Knowledge and Technology

Key Indicators Pre-2010

restructuring(a) 2010–2012(b)

Post-2012 restructuring(c)

Number of internationally published scientific articles

Target(i): 708 Actual(i): 6,835 (cumulative)

Target(i): 708 Actual(i): 6,835 (cumulative)

Target(ii): 1,000 Actual(ii): 1,263 (annual)

Number of graduates from domestic master’s and PhD programs, annual

Target: 240 Actual: 933



Number of technological innovations/process improvements transferred to the productive sector under Component 3

n.a. n.a. Target: 7 Actual: 8

Beneficiary satisfaction level Target(iii): 70% Actual(iii): 94%

Target(iii): 70% Actual (iii): 94%

Target(iv): 80% Actual(iv): 94%

% of annual public R&D funding administered by the ANII

Target: 40.0% Actual: 10.4%

Target: 40.0% Actual: 10.4%

n.a.

Intermediate Indicators Number of research projects receiving direct support




% increase in annual SCI publications per researcher in project-supported research




% increase in annual international coauthored articles per supported researcher




Graduate programs created with support from the project (cumulative)

Target: 6 Actual: 5

Target: 6 Actual: 5

Target: 6 Actual: 5

Graduate programs strengthened with support from the project (cumulative)

Target: n.a. Actual: 14



Number of students completing technical tertiary training supported by the project (cumulative)




Scholarships awarded to young researchers (cumulative)

Target: 175 Actual: 1,132

Target: 175 Actual:1,132

Target: 1,000 Actual: 1,132

Number of Sector Technology Networks established Target: n.a. Actual: 5

Target: 3 Actual: 5

Target: 5 Actual: 5

Number of Alliances supported Target: n.a. Actual: 21



Number of Sector Technology Centers Target: n.a. Actual: 5


Target: 3 Actual: 5

Additional Indicators Collected at ICR Stage Number of registered national researchers Target: n.a

Actual: 1,736 (+189% with respect to 2008) Technical tertiary programs strengthened with support from the project (cumulative)

Target: n.a Actual: 13

Scholarships awarded to graduate students (cumulative)


Evolution of the country's score in higher education and training, as measured by the GII (2007–2015)

Target: n.a. Actual: +18%

adjustments introduced by the restructurings and, on the other side, the improvements in the results matrix to capture such progress.

14




Annual resources (national and external financing) executed by the ANII, US$ million

Target: n.a. Actual: 37 (from 5.5 in 2008)

Note: (a) Targets effective up to the 2010 restructuring. Actual values at project closing. (b) Targets effective up to the 2012 restructuring, according to the 2010 restructuring. Actual values at project closing. (c) Targets at 2015, according to the 2014 restructuring. Actual values at project closing. (i) Revised data source, cumulative. (ii) Revised data source, annually. (iii) Average of: Before award (T0); After award (T1); Implementation (T2); At completion (T3). (iv) Measured at T3, average between entities and individual beneficiaries.

47. The establishment of the ANII and its strengthening were crucial to create better conditions to generate knowledge and technology, through offering more financial and technical assistance and opportunities for knowledge exchange. In particular, 69 percent of the supported researchers and 73 percent of the supported firms declared that they would have not realized their STI activities without support from the ANII (Beneficiary Survey 2013). The increase in the capacity of the ANII to financially support STI stakeholders can be seen in the evolution of the funds executed that, in the period 2008–2014, registered an average annual growth of 44 percent a year, to reach US$37 million in 2014.

48. The evaluations of the Iniciación and Posgrado scholarships offer some insights on the results promoted by these activities. Among the undergraduate students who received an Iniciación scholarship in 2008, 73 percent had completed the bachelor’s program as of 2011 (versus 60 percent of the nonbeneficiaries), 62 percent obtained a research-related employment, and 34 percent of those who were employed before the scholarships declared that their wage increased as a result of the scholarship-financed outcome. Apart from completing graduate education, students receiving a Posgrado scholarship had additional benefits as the production of scientific publications (in 85 percent of the cases) and the possibility to present their own research in conferences and workshops (75 percent of beneficiaries). Nearly 74 percent of the beneficiaries enrolled in the next educational level (PhD or Post Doctoral) after completion of graduate studies. Out of the beneficiaries who had a job as of 2014, 95 percent declared employment in research activities. In 74 percent of the cases, the field of research in which they were currently working was closely related to the research topic developed under the scholarship.

49. The impact evaluations confirm that beneficiaries of the Iniciación and Postgrado scholarship obtained, on average, a higher number of annual publications per year as compared to with respect to nonbeneficiaries and a higher number of publications in refereed journals. Similarly, beneficiaries were more likely to continue their career in the research sector with respect to the control group.22 23 The econometric techniques applied allowed the conclusion that these effects are attributable to the scholarships.24

22 ANII. 2012. Impacto De Becas De Iniciación A La Investigación. Documento de Trabajo n°3. 23 ANII. 2014. Informe de Evaluación Ex-Post, Becas de Posgrados Nacionales. Documento de Trabajo n°8. 24 Both evaluation used panel data and applied a difference-in-difference methodology. The counterfactual was built through Propensity Score Matching among the group of applicants who have not received the scholarship, that ex ante present very similar characteristics to the group of beneficiaries. The external and internal validity of the results are subject to the standard limitations proper of quasi-experiments.

15

50. It is important to highlight that in addition to the activities directly supporting human capital accumulation under Component 2, the support for public-private partnerships for STI (21 Alliances, 3 Stimulating Demand Projects, 3 Networks, and 5 Centers) offered concrete opportunities and demand to generate knowledge and technology specifically tailored to the private sector. Public-private partnerships promoted knowledge and technology generation in strategic sectors such as biopharmaceuticals, ICTs, and agro-business. Examples include research to improve quality of primary sector exports (Networks and Alliances); development of new techniques to process pharmaceuticals (Networks and Alliances); design of new user-friendly platforms to develop ICT prototypes (Centers); and development of cryptography systems to improve safety in online banking (Centers).

Strengthen Uruguay's Capacity to Transfer and Adapt Knowledge and Technology25

51. Rating: Pre 2010 restructuring rating: Modest/ 2010-2012 rating: Substantial/ Post 2012 restructuring rating: High

52. The objective was achieved. Efficacy in achieving this dimension of the PDO has increased from modest to high during the project implementation period, especially thanks to the strategic implementation changes introduced in the 2010 restructuring (TIMBO, Alliances, and Networks) and in the 2012 restructuring (Centers), and the relative changes in the Results Framework. Two out of five key targets effective up to 2010 were surpassed,26 two out of three key targets effective up to 2012 were surpassed; and three out of three key targets effective up to 2015 were surpassed. Overall, results indicate that the Project contributed to create the necessary environment to increase transfer and adaptation of knowledge and technology by the private sector.

Table 4. Results: Strengthening Uruguay's Capacity to Transfer Knowledge and Technology




% of annual variable costs of technology centers generated by the sale of services

Target:10 Actual: n.a.

n.a. n.a.

Number of patent applications submitted by supported research consortia, cumulative

Target: 4 Actual: 1

n.a. n.a.

Number of technological innovations/ process improvements transferred to the productive sector under Component 3

n.a. n.a. Target:7 Actual: 8

Beneficiary satisfaction level in % Target(i): 70 Actual(ii): 94

Target(i): 70 Actual(ii): 94

Target(ii): 80 Actual(ii): 94

% of annual public R&D funding administered by the ANII Target: 40 Actual: 10.4

Target: 40 Actual: 10.4

n.a.

Project-supported researchers introduced into industry Target:40 Actual:44

Target:10 Actual: 44

Target:30 Actual: 44

Intermediate Indicators Articles downloaded from TIMBO (annual) Target: n.a.

Actual: 316,078 Target: 184,134 Actual: 316,078

Target: 262,326 Actual: 316,078

25 This is intended as the country's capacity to transfer existing knowledge and technology (potentially developed in other countries) and to adapt it to a different context in the productive sector. The concept of adaptation of knowledge and technology is sometimes used to indicate adaptation of STI developed abroad to the domestic context despite the concept being broader than this, capturing in general adaptation from one sector to another. In this ICR, we consider the broader definition of adaptation but present specific examples of adaptation of STI developed abroad to the domestic context, to further reinforce the argument supporting project efficacy. 26 It is not possible to measure the actual value at project closing for one of the five original key indicators.

16




Number of informal links created to support Networks Target: n.a. Actual: 36



Number of formal links created to support Networks Target: n.a. Actual: 21



Number of Sector Technology Networks established Target: n.a. Actual: 5

Target: 3 Actual: 5

Target: 5 Actual: 5

Number of Alliances supported Target: n.a. Actual: 21



Number of Sector Technology Centers Target: n.a. Actual: 5

Target: n.a. Actual: 5.

Target: 3 Actual: 5

Additional Indicators Collected at ICR StageIncrease in the university-industry collaboration in R&D, as measured by the relative score in the GII (2007–2015)


Increase in state of cluster development, as measured by the relative score in the GII (2007–2015)


Increase in companies' spending in R&D, as measured by the relative score in the GII (2007–2015)


Increase in firm-level technological absorption, as measured by the relative score in the GII (2007–2015)


Increase in foreign direct investment (FDI) and technology transfer, as measured by the relative score in the GII (2007–2015)


% of firms receiving innovation grants by the ANII, out of total receiving innovation grants from the public sector

Target: n.a. Actual: 50% during 2010-2012 (up from 20% in 2007-2009)

Note: (a) Targets effective up to the 2010 restructuring. Actual values at Project closing. (b) Targets effective up to the 2012 restructuring, according to the 2010 restructuring. Actual values at Project closing. (c) Targets at 2015, according to the 2014 restructuring. Actual values at Project closing. (i) Average of Before award (T0); After award (T1); Implementation (T2); At completion (T3). Revised data source, cumulative. (ii) Measured at T3, average between entities and individual beneficiaries.

53. The information that innovation experts from the World Economic Forum compile each year (GII) allows to appreciate Uruguay's performance in terms of transfer and adaptation of knowledge and technology by firms in the whole period 2007–2015. In particular, table 4 presents results in some key areas directly supported by Project activities.27 Not only did Uruguay score better than the regional average in all these dimensions as of 2015, but the improvement registered in the period 2007–2015 was larger than the one observed for the region overall.

54. The creation and strengthening of the ANII was at the base of the promotion of transfer and adaptation of knowledge and technology among firms. The role of the ANII in this sense increased considerably between 2007 and 2012.28 During 2007–2009, the Ministry of Industry was the leading institution in terms of innovation grants to the private sector. By 2010–2012, this role had been taken by the ANII in an effort to concentrate in this institution both technical and financial support to firms investing in STI. The share of firms obtaining an innovation grant from the ANII (over the total number of firms obtaining an innovation grant from the public sector) increased from 20 percent in 27 The substantial increase of the rates covered in the 2015 version of the GII has likely been capturing the spillovers of this particular project. The sectoralpublic-private partnership for innovation under Component 3 directly target the increase in the university-industry collaboration in R&D, cluseter development, increase in companies' spending in R&D (see also Annex 3). Besides, the limited scale of the Uruguay innovation system makes the Bank intervention quite relevant. 28 ANII. 2015. Encuesta De Actividades De Innovación En La Industria Manufacturera Y Servicios Seleccionados (2010–2012). Colección Indicadores Y Estudios Nr.9. Last data available are from 2012. The figures presented do not include tax deductions but only ‘active’ subsidies/grants.

17

2007–2009 to 50 percent in 2010–2012.29 In terms of internationalization of activities, the agency has signed cooperation agreement with 34 countries in Latin America, North America, Europe, and Asia since 2008.

55. The 1,138 Iniciación scholarships and the 142 research subprojects financed under Component 2 played an important role in directing young researchers toward areas of study that were particularly relevant for the productive sector, thus favoring the creation of knowledge that could easily be adapted to the specific needs of firms. The evaluation of proposals for Sectoral Funds, in particular, was subject to assessment by the Comité de Agencia, a committee integrated by the representatives of the private sector to allow for alignment between research and productive strategy. Posgrado scholarships also played a role: 63 percent of beneficiaries declared that research developed under the scholarship was intended to solve specific problems in the productive sector.30 Forty-four of the researchers supported by the Project were then introduced in the private sector. It is worth mentioning the TIMBO portal as another activity that favored the transfer of knowledge by ensuring universal access to international scientific publications. The number of publications downloaded each year increased from 146,961 in 2009 to 316,078 in 2014 (+215 percent).

56. The Project supported university-industry collaboration in R&D by 31 public-private partnerships for STI, involving 216 firms and all the principal public research institutes of the country (UdelaR; the National Institute of Research in Agriculture and Livestock [Instituto Nacional de Investigación Agropecuaria, INIA]; the Technological Laboratory of Uruguay [Laboratorio Tecnológico del Uruguay, LATU]; ORT University; Universidad de Montevideo; and Instituto Pasteur). Flexible instruments such as Alliances and Networks (starting in 2010) attracted smaller enterprises to be the recipients of transfer and adaptation of knowledge and technology transfers. Thirty-eight percent of firms participating in these instruments are SMEs.

57. Public-private partnerships for innovation also favored the adaptation of knowledge and technology generated abroad to the Uruguayan context. As an example, the Alliance working at the development of non-wheat bread (recycling grains not suitable for export) included the Argentinean National Industrial Institute of Technology, which already had the experience to process this type of flour. Another example is the Alliance patenting preclinical evaluation methods for probiotic strains built on the adaptation and harmonization of biotechnological capacities in France, Spain, and Uruguay. Another interesting example is offered by the Center for ICTs. The Center's work has been crucial to transfer and adapt international knowledge in the field of smart grid management and automatic detection of frauds in e-banking. Interestingly, the Center itself is an example of adapting knowledge and technology, as its design was built by adapting a special flexible organizational model already implemented in France for similar type of technology centers.

3.3 Efficiency Rating: Substantial

58. The Economic Analysis included in the PAD did not provide an estimate of the IRR of the Project. The Economic Analysis in this ICR (annex 3) presents the estimations of the IRR and the

29 The total number of firms receiving public innovation grants increased from 5 percent in 2007–2009 to 8 percent in 2010–2012, which makes the results obtained by the ANII even more meaningful. 30 ANII. 2014. Informe de Evaluación Ex-Post, Becas de Posgrados Nacionales. Documento de Trabajo n°8.

18

benefit-to-cost ratio for the Posgrado scholarships and Alliances; and relevant descriptive information on the TIMBO portal, Iniciación scholarships, Networks and Centers.

59. The creation and management of the TIMBO portal allowed an increase in the number of scientific publication downloads in the country (+215 percent between 2009 and 2014), halving the unitary cost of download in five years, from US$9.78 (2009) to US$4.82 (2014) and thereby increasing efficiency of investment in access to scientific publications.31

60. The estimation of the IRR for Posgrado scholarships considers three scenarios, to test the sensibility of the results to different assumptions on the impact of the program (annex 3). Even in the most conservative scenario, the IRR is estimated to be as high as 21 percent. Considering more optimist assumptions in line with the recent findings from Chile (Solis 2013), IRR would be 40 percent.

61. Firms participating in Alliances were able to increase their revenues (in US$) by 148 percent between 2010 (first year of financing) and 2013 (last year for which data are available). Employment also grew as the number of jobs generated by these firms increased from 1,632 in 2010 to 40,154 in 2013. Comparison with nonbeneficiary firms suggests that at least 77 percent of this employment was generated directly by activities related to the Alliances. The IRR for this activity is estimated to be 38 percent (annex 3).

62. Recent work by Aboal and Garda (2015) for Uruguay suggest that participating in a public-private partnership for innovation is estimated to increase firms' investment in innovation by 52 percent compared to firms that do not participate in such collaboration. Technological innovation output is 77 percent higher among participating firms and nontechnological innovation is 400 percent higher. Crespi and Zuniga (2012) estimate that, for the case of Uruguay, R&D cooperation between firms could increase intensity of investment in innovation per employee by 0.53 percentage points.

63. The Project efficiency is rated as Substantial. The IRR estimations for those activities for which it was feasible to estimate returns are compelling. The additional descriptive information collected and the references from the literature review (annex 3) all support the efficiency of the remaining Project activities. None of the information collected in this ICR seems to indicate any potential wasteful investment nor less-effective investment than what was expected at appraisal. On the contrary, the scale of outputs, compared to initial estimation in the PAD, suggests that the Project may have been more effective than what was expected at appraisal.

3.4 Justification of Overall Outcome Rating Rating: Satisfactory

64. The Project objective was relevant throughout the life of the Project and its design innovative, comprehensive, and suitable to achieve the PDO. The Project outcomes are positively evaluated. All PDO results according to the last restructuring were surpassed, and 20 out of 22 intermediate indicators were mostly achieved or surpassed. Economic efficiency is rated Substantial. The overall

31 The centralization of the system through the TIMBO led, on the one hand, to enjoy a higher bargaining power in the negotiation of business terms and, on the other hand, to expand the scope of benefits among the population.

19

ratings for the three periods are weighted by the disbursement percentages associated with each implementation period, yielding an overall weighted rating equivalent to Satisfactory.

Table 5. Weighted Project Rating

Pre 2010

restructuring2010-2012

Post 2012 restructuring

Overall

Pre- and Post-restructuring outcome rating

Relevance Objectives High High High

Substantial Design Substantial Substantial Substantial

Efficacy

Overall Efficacy Modest Substantial High

Substantial Strengthen Uruguay's capacity to generate knowledge and technology

Substantial Substantial High

Strengthen Uruguay's capacity to transfer and adapt knowledge and technology

Modest Substantial High

Efficiency Substantial Substantial Substantial Substantial

Rating Moderately

Unsatisfactory Satisfactory Satisfactory Satisfactory

Overall outcome rating Rating value 3 5 5 Disbursement weight

23% 45% 32%

Weighted value

0.7 2.3 1.6 5

Final Rating

Satisfactory

3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development

65. At the macro level, a society with more human capital, higher productivity, and falling cost of technology enjoys higher rates of long-term economic growth. Human capital accumulation and economic growth are likely to lead to lower unemployment, higher household income, and lesser poverty.

66. Importantly, although the Project activities did not directly target the poor population, some of the financed subprojects were particularly important for the more vulnerable citizens. In particular, the high social impact subprojects financed research aimed at tackling drug addiction, boosting economic development in smaller towns, and improving women’s health during pregnancy. Additionally, innovative outcomes generated by some Alliances have important medical applications (early detection of viral diseases and easier production of pharmaceuticals), which could be particularly beneficial for the poor.

(b) Institutional Change/Strengthening

67. The institutional change supported has no going back, and it is clearly one of the big achievements of this Project. The new institutional framework allowed for better coordination among ministries (GMI), the development of a national innovation strategy identifying priority areas of action (PENCTI), and the creation of an implementing agency (the ANII) able to execute STI

20

programs in a rapid and efficient manner. The ANII, in particular, reduced fragmentation in public support for STI by streamlining procedures and consolidating competitive funding for innovation from several sectors. The organization of activities of the ANII mirrored the multipillar project design; the decision to intervene both on the supply and the demand side of STI activities proved successful and allowed the ANII to proactively create links between private and public stakeholders and to be better positioned to exploit synergies and avoid overlaps among STI programs. Moreover, the ANII incorporated lessons from project operations into daily activities, starting from procurement arrangements to M&E, leading to enhanced transparency in the management of funds for STI. Overall, the institutional changes supported are evaluated as both sustainable and effective in filling the sectoral gaps highlighted in section 1.1. Activities under Component 1 directly supported the creation and strengthening of the ANII as well as the draft of the PENCTI, setting the basis for an integrated innovation policy.

(c) Other Unintended Outcomes and Impacts (Positive or Negative)

68. Project activities and, in particular, the instruments financed under Component 3, induced a long-awaited change of innovation culture among the private sector. Equally important, the sponsored public-private partnerships had a significant role in eliminating barriers and increasing trust between the academia and researchers on the one hand and firms on the other hand. This outcome has been confirmed both by indicators presented in section 3.2 and interviews conducted during the ICR mission.

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 69. Not applicable.

4. Assessment of Risk to Development Outcome Rating: Negligible to Low

70. Overall, the risk that the strengthened capacity to generate, transfer, and adapt knowledge and technology might not be maintained is Low. The ANII has established itself as the reference institution for implementing STI programs in the country, and it has proven to be able to efficiently deal with program implementation, including adaptation to changing circumstances, hands-on assistance, procurement, and M&E. The National System of Competitiveness Law Bill is proposing to further reinforce the ANII management team by switching from seven voluntary directors dedicating 20 percent of their time to the agency, to three full-time, paid directors (Presidency of Uruguay, March 2015). The strengthening of the role of the ANII and financial support for its programs is confirmed by the 2015–2019 Budget Law Bill,32 which would imply an "increase in the resources devoted to research and innovation. In particular, the Government will increase the investment in STI to be executed by the ANII by Ur$200 million in 2016 and Ur$300 million starting from 2017".33 ANII subprograms developed under the Project will continued to be open and financed by both national and external resources. The Government has negotiated external financial support with the IDB to ensure the flow of resources needed to sustain the financing most of the Project's activities. The IDB's ‘Innovation Program for Productive Development, 2015–2020 will finance 32 At the time of writing this ICR, the 2015–2019 National Budget Plan has been already approved by one of the two Chambers of Government. 33 Presupuesto Nacional 2015–2019, Proyecto Poder Ejecutivo, Mensaje y Exposición de Motivos.

21

scholarships, Sectoral Funds, Alliances, Networks and Centers. Overall, Uruguay is evaluated to have the necessary institutional capacity and financial and legislative commitment to sustain the development outcome.

71. Two important steps will be important in the near future to move the development outcome further. First, the strategic legal framework for innovation policy, the PENCTI, was approved in 2010 and needs to be updated to keep up with national and international trends. Second, a more proactive role by the GMI or relevant political actor will be needed to further strengthen coordination between executive lines and frequently update the strategy of intervention. Importantly, both steps are likely to happen. There is a cross-sectoral consensus among STI stakeholders in Uruguay about the importance of these two actions that have also been recognized in the National System of Competitiveness Law Bill.

5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory

72. The lending instrument was appropriate and the Project's design innovative and comprehensive. Project preparation was informed by examples of good practices and lessons learned from projects in Latin America and other regions, as well as analytical background studies. However, there were one moderate shortcoming. The Bank team underestimated the time needed to develop instruments such as public-private partnerships for innovation, a risk clearly highlighted among the lessons learned from past operational experience in other countries (more details in section 2.1). The PAD did not envisage measures to mitigate this risk, as for instance proactive promotion initiatives among firms and collaboration with local agencies having a strong field presence (Camaras Empresariales, Consejos Sectoriales del Ministerio de Industria). There were, additionally, minor shortcomings related to the Results Framework (more details in section 3.2).

(b) Quality of Supervision Rating: Moderately Satisfactory

73. At least two supervision missions were carried out every year. There was a strong focus on the PDO and outcome indicators as well as fiduciary policies. Regular field visits to subprojects were organized and aide memoires were candid on the agreements between the Bank team and the borrower. Especially important for successful STI projects, the Bank team was proactive in restructuring the Project and adapting activities so as to ensure achievement of the PDO. A midterm review was carried out in 2010, and lessons learned were incorporated. Knowledge exchange events were organized to drive critical activities. In 2012, the Bank team organized a study tour in Spain and Denmark, which was strategic in eliminating the remaining concerns about the creation of Centers. Staff from the Competitiveness Global Practice participated in supervision, technical assistance, and knowledge-exchange activities. The team was also formed by external consultants specialized in trade and competitiveness.

74. There was, however, one moderate shortcomings in the quality of supervision. The many knowledge exchange activities, highly valued by the borrower, could have been organized at an

22

earlier stage of Project implementation. There were also minor shortcomings. Information on progress against indicators contained in Implementation Status and Results Reports and aide memoires were not always consistent. The 2010 Restructuring Paper, as well as the aide memoires of that period, provide little information on the rationale behind some of the changes introduced. The PDO could have been simplified/streamlined.

75. (c) Justification of Rating for Overall Bank Performance

Rating: Moderately Satisfactory

76. Taking into account the ratings of Moderately Satisfactory on both ensuring quality at entry and supervision, the overall Bank performance is rated as Moderately Satisfactory.

5.2 Borrower Performance (a) Government Performance Rating: Satisfactory

77. The Government provided its full support for the Project as shown by the consecutive development strategies that gave high priority to STI. Budget commitment was adequate and stable. Importantly, the Ministry of Economy and Finance (MEF) directly pushed for the Project to keep its focus on the links with the country's productive strategy and the private sector, notwithstanding the early difficulties in disbursement of Component 3. The creation of the GMI provided a first attempt to create a direct link between the Government executive branch and the implementing agency. Nevertheless, there was probably room for better coordination among ministries that would have ensured a more decisive action of the cabinet in setting strategic STI guidelines more frequently and promptly.

(b) Implementing Agency or Agencies Performance Rating: Satisfactory

78. The ANII exhibited strong ownership of the Project and commitment to the achievement of its objectives. The agency was staffed with qualified professionals who demonstrated their experience in dealing with complex issues. Activities under Component 3 had a slow start, but when it was clear that public and private stakeholders needed to be proactively sought and supported, the ANII took appropriate action. M&E and fiduciary arrangements were satisfactory. Overall, Project management was Satisfactory, notwithstanding the difficulty in managing multiple subprojects, components, and collaboration with national stakeholders as well as different international development agencies.

(c) Justification of Rating for Overall Borrower Performance Rating: Satisfactory

79. Based on the ratings on Government and implementing agency performance, the rating is Satisfactory.

6. Lessons Learned

23

80. Strong government support is crucial for quick and smooth implementation, but increased political coordination among executive branches may be necessary to move forward. Strong government support for STI, both in terms of design of the national development strategy and reliable budget allocation, allowed for rapid creation and strengthening of the innovation system and the ANII as the implementing agency of STI programs, practically from scratch. Coordination among ministries could be improved in the future by designing better incentives for a more proactive role of the ministries in defining and updating the STI strategy (see section 4). Importantly, it could be worth discussing to have one political institution that takes responsibility to coordinate at the strategic and political level in the area of STI (for instance, the Ministry of Science, Technology and Productive Innovation, as in the case of Argentina) or simply a more direct link between the GMI and the presidency.

81. The creation of a single agency responsible for the implementation of STI programs improves coordination, efficacy, and efficiency of the STI policy. The ANII represents a successful example in this sense. The quite ‘light’ and flexible organizational structure of the ANII, the choice to rely on a relatively small but highly qualified pool of human resources, and the constant learning process from international cooperation, allowed for a quick start of operations, implementation of programs in line with international standards, and the achievement of significant results in terms of STI in a relatively short period. A new design of the ANII's Board (see section 4) would further smooth the decision-making process.

82. It is a good practice for STI projects to address and integrate different pillars of the STI system, by including actions that affect both demand and supply of knowledge and innovation. In the case of this Project, the activities simultaneously promoted institutional strengthening, human capital accumulation, and innovation in the productive sector. This complex Project design was successful in building the main pillars of an innovation system in the country. The integration of different mechanisms to promote STI since the early foundation of the system allows for positive spillovers between activities, increased knowledge exchange opportunities, reduced operative costs, setup of a smooth system for continuous learning, and early detection of opportunities. The design of STI policy as an organic system sets the basis for a natural upgrade of the STI policy as a key factor in the National Competitiveness System.

83. The Project design should envisage early proactive action by the implementing agency to develop a partnership culture and demand, especially among the private sector. This could be achieved by stronger coordination with other government local agencies/programs that have a strong field presence and frequent contacts with the private sector (in the case of Uruguay, Camaras Empresariales, Consejos Sectoriales del Ministerio de Industria, Oficina de Planeamiento y Presupuesto).

84. In designing a results matrix for STI projects, teams should consider the long term usually associated with the development of innovation outcomes, as well as the country-specific context that is likely to have a direct impact on STI through different channels—the education system, the level of international integration, and the industrial strategy. It is particularly challenging to design an accurate results matrix for STI projects. There is no clear international standard, which would be the most appropriate indicator for STI outcomes (rather than inputs). Furthermore, it is difficult to combine international guidelines with country specificities, given the many factors that directly play a role in determining STI outcomes. Finally, outcome indicators (for

24

instance, development of innovative products and services, increase in productivity, and trade in R&D services) may be difficult to materialize in the relative short life of a project (usually five years), especially if the innovative system has to be built from scratch, which was to some extent the case in this Project. The discussion on STI indicators is ongoing and recently published work (OECD;34 National Academies 2014;35 EU 201236) can be of help in designing a results matrix for future STI operations, both to incentivize implementation of activities on time and evaluate achievement against objectives.

85. It may be especially important for the task team and the implementing agency to rapidly incorporate early lessons and changed circumstances. Although this is true for all kinds of operations, it may be even more relevant for the STI sector, which is at the same time relatively new and rapidly evolving in developing countries. In the case of this Project, the relatively flexible design of the activities and disbursement categories on the one hand and a very proactive approach by the implementing agency and the task team on the other hand, allowed to provide prompt and adaptive responses to ensure achievement of the PDO.

86. All these lessons learned have general applicability for similar operations in the same subsector.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies

87. Not applicable.

(b) Cofinanciers

88. Not applicable.

(c) Other partners and stakeholders

89. Not applicable.

34 OECD. Main Science and Technology Indicators. 35 National Academies. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. 36 EU (European Union). 2012. Research Infrastructures for The Assessment of Science, Technology, and Innovation Policy.

25

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in US$, million equivalent)

Components Appraisal Estimate

(US$ million)

Actual/Latest Estimate

(US$ million)

Percentage of

Appraisal Total Baseline Cost 32.5 32.5 100.0

Physical Contingencies 0.0 0.0 0.0Price Contingencies 0.0 0.0 0.0

Total Project Costs 1. Support STI Policies and Institutional Strengthening 3.1 9.3 299.71.1 Support the development of public STI policies 0.8 0.2 27.51.2 Support National Research and Innovation Agency 1.8 9.0 501.7Support for TIMBO // 7.1 N/A1.3 Setup and Support STI Observatory 0.5 0.0 8.02. Build human capital and high-quality research teams 16.7 18.3 109.72.1 Promote excellence in science and technology research in priority areas 5.3 4.6 86.82.2 Increase the stock of advanced human capital 9.6 12.0 125.42.3 Strengthen priority technical tertiary education programs 1.8 1.7 93.93. Promote Innovation and Technology Transfer to the Productive Sector 11.5 4.5 39.03.1 Develop sector technology centers 4.3 2.3 53.63.2 Support for public-private research consortia 7.3 2.2 29.8Unallocated 0.8 0.0 0.0Front-end fee PPF 0.5 0.4 80.0Front-end fee IBRD 0.0 0.7 N/A

Total Financing Required 32.5 32.5 100.1 (b) Financing

Source of Funds Type of

Cofinancing

Appraisal Estimate

(US$ million)

Actual/Latest Estimate

(US$ million)

Percentage of Appraisal

Borrower – 6.50 6.50 100.00International Bank for Reconstruction and Development

– 26.00 26.00 100.00

26

Annex 2. Outputs by Component

Component 1. Support STI Policies and Institutional Strengthening

1. The PDO indicator selected to monitor the achievement of this component was defined as 80 percent of beneficiaries were satisfied as measured through perception surveys. This indicator was surpassed: satisfaction level was 92 percent among individual beneficiaries and 95.5 percent among entities benefitting from the ANII's programs. Among other things, the survey inquired on the suitability of the financing instrument used to achieve the individual's/firm's STI objective (reflecting to some extent suitability of STI policies and programs) and the satisfaction with the support received by the ANII at various stages of subprojects implementation (capturing institutional strengthening).

Component 1 - Summary of Outputs Indicator* 94% is the beneficiaries' satisfaction with the ANII's performance and suitability of programs.

KI

1.3 months is the reduction in the average time it takes for the ANII to process grant proposals between closing of the call and final selection, compared to 2011.

II

28 program evaluation studies (among which, 5 impact evaluations) performed or contracted by the ANII

II

35 STI monitoring indicators tracked annually based on reliable methodologies II 316,078 articles downloaded from TIMBO (215% increase from the portal launch in 2009) II Elaboration of the PENCTI ICR US$162 million, including national budget resources and external financing, were executed by the ANII during 2008–2014.

ICR

80 projects processed on average by each ANII official, per year (2013) ICR 40 external financial audits, all approved with no material observations (2014) ICR 73% of the beneficiary firms would not have invested in innovation without the ANII's support (2013 survey to beneficiaries).

ICR

Note: * KI = Key Indicator; II= Intermediate Indicator; ICR = Data collected during the ICR stage.

2. This component was strategic to ensure the success of the Project as it directly supported the significant institutional strengthening that was the necessary condition for a coherent and effective implementation of innovation policies. In 2001, there had been an attempt to institutionalize innovation policy with the establishment of the DINACYT as a department within the MEC. The DINACYT assumed responsibilities for coordinating, implementing, and evaluating programs related to innovation. In addition, the National Advisory Council on Innovation, Science, and Technology (CONICYT) was created in 2001 to act as an advisory committee to the Government suggesting policy guidelines. This institutional framework had not proven effective. The MEC had been moderately successful in supporting basic science but little progress had been made about stimulating public-private research linkages and innovation. The DINACYT’s funding had been volatile, and the implementation of innovation programs remained dispersed among other agencies.

3. This component financed four main sets of activities. Activities under Subcomponent 1.1 (Support the development of public STI policies) included national consultancies and workshops to draft strategic documents and promote critical discussion among key stakeholders on STI policy issues. Although these activities were carried out throughout the entire Project implementation period, they were particularly frequent in the initial stage, consistent with the need to set up the policy framework and organizational structure required for implementation of the Project itself.

27

Consultancies and workshops focused on four main topics: (a) identification and design of instruments to improve synergies and collaboration between public and private stakeholders; (b) periodic inventory of the national research capacity (research groups, update of the registry of researchers, innovation capacity in the private sector, and so on) and the submission of proposals to improve such capacity; (c) mapping of interaction opportunities with Uruguayan researchers and entrepreneurs based in other countries, to carry out specific activities financed and promoted by the ANII; and (d) development of training activities to improve the management of public and private research and technology institutions, as well as the design of public STI policy.

4. All these activities came together to form the basis for the elaboration of the PENCTI, approved by ministerial decree in 2010. The set of objectives presented in the plan was highly influenced by the Project's design, as it aimed at simultaneously building critical STI pillars such as (a) strengthening the STI system and its link with the productive strategy; (b) improving private sector competitiveness; (c) improving opportunities for inclusive access to knowledge and stimulating research on social problems; (d) strengthening human capital; and (e) building a strong M&E system on STI indicators and policies.

5. Importantly, the PENCTI identifies the key areas to be targeted by investment in innovation. It allows for improved efficacy and efficiency of STI investment, as well as its alignment with the country's productive strategy. The PENCTI identifies (a) ICT; (b) biotechnology; (c) specific sectors with high potential (as nanotechnology) as technological areas to be prioritized; (d) software; (e) human and animal health; (f) agriculture and livestock production; (g) environment; (h) energy; (i) education; (j) transport and logistics; and (k) tourism, as key sectors presenting special innovation challenges and opportunities.

6. Activities financed under Subcomponent 1.2 (Support the ANII) included training offered to the ANII staff (including courses on management of technology and innovation and the elaboration of business intelligence to ensure efficient and transparent implementation of innovation and research programs); studies; purchase of office equipment; travel costs linked to Project implementation; and dissemination activities. Activities (especially training, studies, and purchase of equipment) were organized around three main objectives: (a) to develop internal and external evaluation systems to assess the use of resources and the results achieved by the ANII; (b) to build a strong human resources system that is able to attract qualified professionals, ensure transparent hiring procedures, and evaluate the ANII officials’ performance; (c) to develop an IT system and an institutional website to facilitate management of activities and external communications, respectively.

7. These activities, combined with technical assistance throughout the Project's duration, contributed to the creation and strengthening of the ANII. The agency executed US$162 million between 2008 and 2014 and has gone through 40 external financial audits, all approved with no material observation. It currently has 54 permanent staff, 7 interns and 7 consultants. Nearly two-thirds of the staff have completed tertiary education and the remaining one-third is currently enrolled in graduate programs.

8. Each ANII official processes, on average, 80 STI projects per year. The average time to process grant proposals between the closing of the call for proposals and final selection (Intermediate Indicator 1, 76% achieved) reduced from 5.7 months in 2011 to 4.4 months in 2015 but would be reduced to 3.5 months if one excludes Sectoral Funds. This is because the selection of Sectoral Funds

28

involved an external committee composed of research institutes and firms (Comité de Agencia). Although the Comité represented a further step in the selection process, it brought a high added value in terms of aligning research with the productive strategy. Furthermore, a general positive trend may be observed among those funds awarded more than once. The evaluation time fell by 110 days for the Agro Fund (from 353 days in 2010 to 243 days in 2014); by 53 days for the Fishery and Aquiculture Fund (from 219 days in 2013 to 166 in 2014); and by 216 days for the Energy Fund (from 343 days in 2010 to 127 days in 2014). A slight increase is observed only for the Digital TV Fund (from 75 days in 2012 to 92 days in 2014).

9. The 2010 restructuring increased the scope and spending of Component 1 to include the financing of the TIMBO portal. The TIMBO portal was developed by the ANII with the objective of ensuring universal and free-of-charge access to international scientific and technological bibliography and patent databases.

10. The Uruguayan TIMBO has been recently recognized as best practice by the GII (2015) (more information available in this info graphic available at <http://www.wipo.int/export/sites/www/pressroom/en/documents/gii_2015_infographic2.pdf>). The TIMBO portal allowed for ‘democratization’ of scientific knowledge, reduced the need for universities and research institutes to act as intermediaries (therefore decreasing the burden on their budget), expanded the scope of the target audience, and ultimately increased the number of publication downloads and decreased unitary costs. In 2014, 316,078 articles were downloaded from TIMBO (Intermediate Indicator 4). More information on the TIMBO portal is available in this video available at <http://www.timbo.org.uy/tutoriales/10> (in Spanish).

11. Activities financed under Subcomponent 1.3 (Set up and support the STI Observatory) included training of staff in the ANII M&E unit, studies, dissemination events, development of management information systems, purchase of office equipment, and travel costs of the M&E unit officials. In particular, training activities had a clear international footprint that allowed the ANII to develop a strong M&E system in a relatively short period, as a result of lessons learned from international experience, and be part of international networks on this topic (especially the RICYT) favoring continuous quality improvement and sustainability of M&E activities.

12. The M&E unit of the ANII has performed or contracted 28 studies since its creation (technical and sectoral reports, surveys, and qualitative and quantitative evaluations) (Intermediate Indicator 2, 140 percent achieved), among which 5 are impact evaluations. Data for an additional impact evaluation on Sectoral Funds is being collected at the time of writing this document. The M&E unit annually tracks 35 STI monitoring indicators (Intermediate Indicator 3, 100 percent achieved), including STI input indicators related to human resources and financing, output indicators of scientific production, indicators of innovation in the private sector, and indicators on STI culture.

Component 2: Build Human Capital and High-quality Research Teams

13. The PDO indicators selected to monitor the achievement of this component were defined as 1,000 internationally published scientific articles; 600 graduates from domestic master’s and PhD programs; and 30 project-supported researchers introduced into industry. All the indicators were surpassed. In 2014 alone, 1,263 scientific articles were published internationally (6,835

29

cumulatively). In 2014, 933 students graduated from domestic master’s and PhD programs (4,167 cumulatively); 44 supported researchers were introduced into industry at the end of the Project.

Component 2 - Summary of Outputs Indicator* 6,835 internationally published scientific articles in the period (cumulatively, 1,263 in 2014)

KI

933 graduates from domestic master’s and PhD programs KI 142 research projects received direct support II 33% increase in annual SCI publications per researcher in project-supported research II 25% increase in the annual number of international coauthored articles per researcher in project-supported research teams

II

5 graduate programs created with support from the Project II 14 graduate programs strengthened with support from the Project II 1,138 scholarships awarded to young researchers II 78.1% is the completion rate of Project-supported students in the program time frame II 200 completing technical tertiary training supported by the program II 13 technical tertiary programs strengthened with support from the Project ICR 23% of students obtaining a PhD during 2008–2014 received support from the ANII ICR

Note: * KI = Key Indicator; II= Intermediate Indicator; ICR = data collected during ICR stage

14. Subcomponent 2.1 (Promote excellence in science and technology research in priority areas). Under this subcomponent, Sectoral Funds provided direct financing to R&D subprojects in priority areas of interest. Strategic areas were identified using the PENCTI: software, IT services and audiovisual production; human and animal health; agriculture and livestock production; environment; energy; education and social development; logistics and transport; and tourism. A special financing line was provided to subprojects likely to have a high social impact. In both cases, eligible beneficiaries were research groups of excellence working in public or nonprofit institutions. Preference was given to multidisciplinary and inter-institutional teams. Some call rounds favored participation of young researchers in the teams. Financing to approved subprojects included research inputs and material, salaries for researchers associated with the subprojects, research experience in a foreign institution, and expenses associated with dissemination of results.

15. The Project provided direct support to 142 specific research activities in strategic areas (Intermediate Indicator 5, 1001 percent surpassed). Sectoral Funds financed 124 research subprojects in agro-industry, fishery and aquaculture, digital television, energy, education, and human and animal health. The line of High Social Impact Projects financed 18 additional research subprojects on topics such as road safety, healthy pregnancy, drug addiction, and economic development in small towns.

16. Subcomponent 2.2 (Increase the stock of advanced human capital). Under this subcomponent, the Project supported the creation and strengthening of domestic graduate programs, as well as scholarships for undergraduate students willing to get professional experience in a research institution, and scholarships for master’s and PhD students for the completion of their graduate studies.

17. Support to national graduate programs included financing of activities such as the preparation of the curricular design, training of teachers and researchers, international mobility of teachers and researchers, purchase of equipment, and salaries of administrative staff. Applicants could solicit financial support up to U$S40,000 per year, in the case of strengthening existing graduate programs,

30

or U$S150,000 per year, in the case of creation of new programs. Financing could be renewed for one year, implying an overall maximum period of 24 months. Benefitting institutions were required to cofinance the creation or strengthening of the programs in an amount that had to be at least equal to the financing provided by the ANII.

18. Five brand new graduate programs were created (Intermediate Indicator 8, 83 percent achieved), including master’s in strategic areas such as Management of Technology Firms (ORT University), Engineering (ORT University), Family and Community Medicine (Montevideo University), and Alimentary Technology (UdelaR). Fourteen graduate programs were strengthened with Project support (Intermediate Indicator 9, 67 percent achieved). Examples include the strengthening of master’s in Chemical Engineering, Environmental Science, Bioinformatics, Mathematical Engineering, and Integrated Cost Management of the Southern Cone, Clinical Pharmacology; and of PhD programs such as Electrical Engineering, Sciences of Agriculture and Cattle, and Biological Chemistry.

19. Another key activity supported by this subcomponent was the award of scholarships to undergraduate students willing to get a first professional experience in a research institution (Becas de Iniciación a la Investigación) and scholarships to master’s and PhD students for the completion of their graduate studies (Becas de Posgrado). Scholarships were awarded for a maximum period of 12 months in the case of Iniciación a la Investigación and for master’s students and 36 months for PhD students. Financing included tuition fees and a monthly income support (up to a maximum amount equal to US$350 per month in the case of Iniciación a la Investigación scholarships, US$700 per month for master’s students, and US$900 per month for PhD students).

20. Overall, the Project financed 971 scholarships to master's and PhD students and 1,138 scholarships for young researchers (undergraduate level) willing to gain experience in a research institution (Intermediate Indicator 10, 114 percent achieved). Seventy-eight percent of these students completed the academic program in the allotted time frame (Intermediate Indicator 11, 97 percent achieved).

21. Several Project activities—among which, direct financial support for research (Sectoral Funds and High Social Impact), the strengthening of domestic supply of graduate programs, and easier access to scientific publications (TIMBO)—contributed to improve the performance of supported domestic researchers in terms of international scientific publications. With respect to 2010, researchers benefitting from Project activities increased by 33 percent the number of annual SCI publication per researcher (Intermediate Indicator 6, 121 percent achieved), and by 25 percent the number of annual international coauthored articles per researcher (Intermediate Indicator 7, 103 percent achieved).

22. Subcomponent 2.3 (Strengthen priority technical tertiary education programs). This component financed the creation or strengthening of technical tertiary programs. Applications were admitted from institutions in both the public and private sector, including nonprofits. Preference was given to applications aiming at developing the supply of technical tertiary programs in strategic areas as identified by the PENCTI. Solicitants could request a maximum amount of US$200,000, and a varying cofinancing was required to the beneficiary institution, depending on the call edition. Financing was directed toward design of the curricular program, training and mobility of teachers

31

and technicians, purchase of equipment, human resources in charge of the administration of the program, scholarships for interns supporting the program, and small infrastructure interventions.

23. Overall, the Project financed the creation and strengthening of 13 technical-tertiary STI programs, among which are Design of Sustainable Tourism, Navigation and Fishery, Intensive Vegetable Production, Agro-Energy, Alternative Energy and Energy Efficiency, Industrial and Mechanical Maintenance, Computer Science; and Biotechnology. Stakeholders' opinion on the ANII'S support for the Biotechnology program is available in this video at <https://www.youtube.com/watch?v=C4g_TzOqiDQ>. A total of 200 students benefitted from the creation and strengthening of such technical tertiary programs (Intermediate Indicator 12, 100 percent achieved).

Component 3: Promote Innovation and Technology Transfer to the Productive Sector

24. The PDO indicator selected to monitor the achievement of this component was defined as ‘seven technological innovations/ process improvements transferred to the productive sector through Centers, Networks, Stimulating Technological Demand projects, and Alliances’. The indicators were surpassed. Examples of innovations/process improvements transferred to firms include system of molecular diagnosis of viral epidemics such as dengue and H1N1; a new technology for the preparation of veterinary vaccines; development and market analysis of a ‘composed flour’ that recycles non-wheat grains not suitable for export (such as soy and oak); quality control of national production of milk; and a new technology for the quality control of biopharmaceuticals.

Component 3 - Summary of Outputs Indicator* 8 technological innovations/ process improvements transferred to the productive sector through Centers, Networks, Stimulating Technological Demand Projects, and Alliances

KI

US$712,883 is the private sector cash contributions to supported Alliances II 36 informal links (without written contract) created to support Sector Technology Networks

II

21 formal links (with written contract) created to support Sector Technology Networks II 5 operational Sector Technology networks Established II 21 operational Alliances supported II 63 entities/companies participating in the Alliances supported II 3 stimulating technological demand projects II 5 operational sector technology centers II

Note: * KI = Key Indicator; II= Intermediate Indicator.

25. Activities under Component 3 supported the creation of different types of public-private partnership for STIs. In order of increasing financial and organizational commitment, these were identified as Alliances, Stimulating Technological Demand projects, Networks, and Centers.

26. Alliances represented flexible research consortia. They were created to find a solution to a specific innovation demand, and dissolved thereafter (Subcomponent 3.2: Support public-private research consortia). Grants to Alliances permitted the financing of the purchase of goods by the beneficiary institutions (general equipment, specific laboratory equipment, interventions to improve health and safety in the workplace, software, and input material); infrastructure (building alterations to allow installation of equipment); training (within the country or abroad); consultancy services (to solve administrative and technical issues); costs associated with intellectual property rights; and

32

operating costs. Solicitants could request a subsidy from the ANII in the amount of up to US$200,000. Beneficiaries were requested to cofinance at least 30 percent of the total cost of the Alliance in cash.

27. Examples of Alliances' objectives include the production of a kit to detect contaminant proteins (see beneficiaries' testimonials at <https://www.youtube.com/watch?v=vPQVVqYF3fQ> and at <https://www.youtube.com/watch?v=Td32o8H3qvw>) or the alimentary use of composite (non-wheat) flours (see <http://www.bancomundial.org/es/news/feature/2014/08/12/innovacion-panaderos> for the Alliance description and results). Sixty-three entities (of which, 43 firms) (Intermediate Indicator 20, 110 percent achieved) contributed with a cash investment of US$712,883 (Intermediate Indicator 15, 142 percent achieved) to a total of 21 Alliances (Intermediate Indicator 19, 123 percent achieved).

28. Networks implied a more long-term relationship. The Network benefitting from the grants offered by the ANII had a maximum of four years to implement the approved business development plan, and arrangements had to be sustainable in the medium and long term, even after completion of a period of ANII support. The Network had to be constituted as a legal entity, and it was the direct beneficiary of the subsidies awarded by the ANII. These grants could be up to US$1 million, representing a maximum of 70 percent of the total cost of the Network. Disbursement categories were similar to those described for Alliances.

29. The Networks put together actors from the private and public sector, which have a significant degree of involvement in productive activities in a common value chain. Unlike Centers, the Networks’ services were offered to Network members only and were set up to address innovation challenges that are specific to the Network. Preference was given to Network proposals that were oriented to the international market.

30. Activities carried out by Networks aimed at identifying technological bottlenecks encountered by the productive sector; finding solutions to overcome such bottlenecks by R&D activities; designing and implementing technology transfer activities among firms; training qualified human resources with sector-specific skills; establishing channels of communication and interaction among Networks’ participants; and investing to create or improve the supply of technological services to the productive sector.

31. The progressive approach of the ANII to increase awareness of this instrument among firms and research institutions resulted in 36 informal links (without written contract) (Intermediate Indicator 16, 100 percent achieved) and 21 formal links (with written contract) (Intermediate Indicator 17, 95 percent achieved) between public and private actors interested in supporting Sector Technology Networks and in the operationalization (at least first disbursement completed) of five Networks (Intermediate Indicator 18, 100 percent achieved). The results of the financed Networks include the promotion of tourism in the Piriapolis Region; the creation of the business incubator Ingenio (a video presentation can be found at <https://www.youtube.com/watch?v=q4bkbJcxDEw>); the characterization of the national production of milk and dairy products; improvements in fruit exports; and the genomic characterization of cattle.

32. Between 2012 and 2014, the Project financed private joint ventures between firms that had a common innovation objective. Three Stimulating Technological Demand projects were approved and financed (Intermediate Indicator 21, 100 percent achieved). The objectives of the three subprojects

33

included the design and production of a special glass container; a center for selling handmade national furniture; and the development of a new heating system. However, during implementation, the instrument was used not as a partnership but simply to finance the acquisition of existing technology from firms, with no generation or adaptation of knowledge and technology and very limited transfer that could be financed otherwise. Given the low justification for public sector subsidies for this instrument, the activity was closed down in 2014.

33. In the period 2012–2015, the Project financed the operationalization of five Sector Technology Centers (Intermediate Indicator 22, 167 percent achieved). The Centers are legal institutions offering services to firms and other organizations in a particular sector. They draw on sectoral experts who help companies/organizations identify their specific demand for technology and develop a related action plan. Interestingly, the Centers build on the early experience encountered by the ANII when promoting the instruments of Component 3 among the private sector. Indeed, very often firms are not aware of their specific innovation demand or cannot identify it clearly. This is particularly true among SMEs. The Centers personnel draw on their expertise to assist firms during this initial, crucial stage of the innovation process.

34. Some of the possible activities of a Center include facilitating the transfer of existing technologies (off-the-shelf) to Uruguayan enterprises; addressing sectoral quality issues; identifying bottlenecks and opportunities for further innovation of products and processes at the sectoral level; conducting R&D to develop such products and processes; facilitating the commercialization of a new product; and collaborating with other research centers and existing institutions. Importantly, the associative nature underlying the creation of implementation of Centers, involving several public institutions and private firms, allows to have both an advantage at the time of generating international collaboration as well as to promote decentralization of activities behind the city of Montevideo. Because of their direct, frequent contact with firms and the deep knowledge about their innovation needs, the Center personnel can help in promoting the financing instruments offered by the ANII to the private sector. Finally, the Centers produce technical publications and regularly organize knowledge exchange events, thus contributing to the generation of knowledge and technology.

35. As defined in the 2014 Project Restructuring Paper, a Center would be considered operational when it meets at least three out of four of the following formal requirements, and two out of four of the following operational requirements:

Formal requirements:

Signature of the contract establishing the Center Center defined as a legal entity Definition of governance by participating stakeholders Development of a working plan/definition of the portfolio together with the

corresponding sector

Operational requirements

At least one assignment (contract) with the productive sector Completion of first disbursement Procurement of equipment (if specified in the application for the grant)

34

Hiring technical personnel (if specified in the application for the grant)

36. The effort to relaunch Centers (2012) was particularly successful, with five operational Centers in only 18 months (2013–2015). The 5 Centers have been endorsed following a competitive selection process, with a strong leadership from the private sector, and anchored in the best resources of the national innovation system of Uruguay. These projects bring together large, small and medium-sized companies, encouraging collaboration between competing companies. The Centers have a similar system of governance, in several aspects, to those of the most successful technology centers in the world, where the private sector leads and has a share of resources for operation.

37. Examples of operational Centers supported by the Project are the Center of Innovation in Engineering (bringing together firms and the engineering faculties of domestic universities not only to meet the specific engineering needs in the private sector but also to promote entrepreneurial culture among engineering students); the Center of Information and Communication Technology (a multidisciplinary center for research and innovation in the field of ICT and its applications for vertical sectors, such as energy, agriculture, banking and health); Unit of Innovation in Alimentary Technology (a center collaborating with similar institutions in Spain, Costa Rica, Argentina and Angola, to promote innovation in alimentary technology among firms); and the Center of Industrial Extensionism (offering innovation services to SMEs in the industrial sector. A video presentation is available at <https://www.youtube.com/watch?v=4JdUr70thik>). Besides the five operational Centers, three more are currently being discussed and approaching the contract signature stage

35

Annex 3. Economic and Financial Analysis

1. Since the creation of the ANII, innovation and research activities have had a formal institutional support, which was nonexistent until then. The aim of this document is to evaluate the social returns to some of the crucial activities financed by this Project through the ANII, namely postdoctoral scholarships (Becas de Posgrado) (Component 2) and Alliances (Component 3)37. Furthermore, the analysis presents specific cost effectiveness information on the TIMBO portal (Component 1), and a short literature review that helps evaluating the efficiency of other main Project activities, namely Becas de Iniciación a la Innovación (Component 2) and Sector Technology Networks and Centers (Component 3)38.

2. For the evaluaion of Becas de Posgrado program, since information about the direct social benefits of scholarships was not available, estimations of potential returns to higher education were calculated using the Encuesta Continua de Hogares (ECH), the main Household Survey for Uruguay. Regressions based on a simple version of the human capital model were estimated for 2006 – 2013, considering the evolution of the coefficient associated with postgraduate education as a proxy of the educational IRR for this population. Additional descriptive data from the scholarships from the ANII were summarized.

3. For the evaluation of the Alliances program, a difference-in-difference model was estimated. Information about revenues for beneficiaries and nonbeneficiary firms, before and after the program, was provided by the ANII. The before and after revenues for treated and nontreated firms were compared in order to estimate the potential (non-conditioned) effect (incremental gains) of the program.

Results by Component

Component 1

4. One crucial aspect promoted by the ANII was the possibility of enabling Uruguayan citizens to have access to bibliographic databases containing high-quality international scientific journals. This was achieved by the setup of the online platform TIMBO. The website provides free access to journals and databases of references among another useful sources of information for R&D activities (Informe de Seguimiento de Actividades 2014).

5. Table 3.1 summarizes the evolution of articles downloaded for each year (from 2009 to 2014), its annual and cumulative variation. One can observe an increasing trend of downloaded documents over time, especially significant in the first years of implementation of the Project and the last available year (2014). Importantly, an increasing number of researchers are pointing at the download of scientific articles as a good indicator of the capacity of publications to generate scientific activity, implicitly recognizing the role of scientific articles download in promoting transfer and generation of knowledge (Bollen et al. 2005, 2008 and 2009; Shepherd 2007).

37 The two activities together account for 32% of total project investment. 38 The information at disposal did not allow for a quantitative estimation of the monetary benefits arising from these activities.

36

Table 3.1. Evolution of Downloaded Articles from the TIMBO Platform Website, Annual and Cumulative Variation (in %)

Year Downloaded

Articles Annual

Variation Cumulative Variation

2009 122,901 0% 0% 2010 191,651 56% 56% 2011 241,401 26% 96% 2012 258,761 7% 111% 2013 274,405 6% 123% 2014 316,078 15% 157% Total 1,405,197

Source: TIMBO, as of September 2015.

6. One of the most promising results is the progressive reduction of the average unitary cost by downloaded article in the period, from an average cost of US$9.78 in 2009 to US$4.82 in 2014. This is a clear improvement in the efficiency of the system, since it was able to reduce costs while maintaining a stable annual investment (table 3.3).

Table 3.3. Average cost per downloaded article and total investment (US current dollars)

Year Average Cost per Article Investment - TIMBO

US$ 1,520 2009 9.78 1,728 2010 8.88 2,265 2011 7.74 1,859 2012 7.32 2,036 2013 7.02 1,749 2014 4.82

Source: TIMBO, information from September 2015.

Component 2

Posgrado scholarships

Descriptive Analysis

7. In the period 2008–2014 a total of 23 percent of doctoral candidates in the country were financed by the ANII. Apart from completing graduate education, students receiving a Posgrado scholarship had additional benefits as the production of scientific publications (in 85 percent of the cases) and the possibility to present their own research in conferences and workshops (75 percent of beneficiaries). Nearly 74 percent of the beneficiaries enrolled in the next educational level (PhD or Post Doctoral) after completion of graduate studies. Out of the beneficiaries who had a job as of 2014, 95 percent declared employment in research activities. In 74 percent of the cases, the field of research in which they were currently working was closely related to the research topic developed under the scholarship. The rate of successful completion of the grants has been high.

37

Estimation of the Social Returns of Posgrado Scholarships

Main Assumptions

i. In absence of information on the distribution of pre-2012 investment by single year, information for the period 2012-2015 was used to derive the average unitary investment per beneficiary, and the number of beneficiaries in each year was considered to distribute the variable part of the pre-2012 investment. The remaning investment can be thought as fixed cost, and it was assumed to be disbursed in the first year of implementation of the program (2008).

ii. Due to the impossibility to distinguish between the salaries of master's and PhD graduates, we assume that on average the benefits of a graduate degree start materializing after 2 years of entry in the program. The duration of a master’s program is two years in Uruguay and the duration of a PhD is four years. This assumption takes into account the high prevalence of students engaged in the labor market at the time of study. According to the University Census of Students - 2012, 88.7 percent of the postgraduate students work in Uruguay (including PhD students) and work weekly 39.2 hours on average.

iii. It is assumed that the differential return to postgraduate education with respect to graduate education stays constant across the working life. This is a medium scenario, as economic, institutional and social changes may cause this relationship to be non-linear over time, and the incremental return to posgraduate education may either increase or decrease with age.

Methodology

8. Following Belli et al. (1998), Mincer equations were estimated for 2006–201339. The Mincer equation allows estimating the postgraduate level return with respect to lower educational levels. Once estimates of the coefficient reflecting the incremental return of a postgraduate student were obtained, the potential per capita benefit from the program was estimated and applied to the postgraduate scholarship holders for each remaining year of working life.

9. The procedure could be summarized as follows:

(a) The Mincer equation allows estimating the return to postgraduate level compared to complete university and the other lower educational levels. The Mincer equation including the level of education, differentiating postgraduates from other educational levels, was estimated using the National Household Surveys (Encuestas Continuas de Hogares) for 2006–2013. Given that the average age of beneficiaries at the time of obtaining the Posgrado scholarship is 29 (ANII, Informe de Seguimiento de Actividades 2014), and based on assumption ii), the differential return of postgraduate education to graduate education was estimated at the age of 3140. Mincer equation could be specified as follows:

39 Last year for which the National Household Survey is available. 40 Mincer equation specifies a MCO regression model where the log of hourly wage is the dependent variable and independent variables are dummies identifying the sex and the race of individuals, and variables that reflect the level of education, sector of activity, and the work experience.

38

In addition to the educational levels and potential experience variables, the estimation considered sex, race, and occupational sector as control variables.

(b) The coefficient of posgraduate education was estimated to be approximately 5. Results suggest that the incremental return of a postgraduate worker over a graduate worker is 0.6641

(c) Using the National Household Surveys, the average annual income among workers who achieved postgraduate level and the average annual income among workers with completed graduate education were estimated. The difference in average annual income is combined to the estimate of the incremental return to approximate the per capita benefits from the program.

(d) This per capita income is applied to each postgraduate scholarship holder, for each remaning year of his or her working life42.

(e) The sum of the benefits for each cohort in each year provides us with an estimate of aggregate benefits per year.

Results suggest that this program enjoys an IRR of 61%. To better get a sense of the program efficiency, we calculate the cost benefit ratio considering two scenarios. In the first one, we apply the social discount rate of 12% estimated by Oddone (2011) for the case of Uruguay. In the second one, a low discount rate of 5% is applied. Table 3.4 presents the results of this exercise. For each US$ invested in this activity, 16 US$ are generated (or 26 US$ in the second scenario).

Table 3.4. Estimation of the Cost-Benefit ratio (Impact on enrollment=100%) Discount rate 12% 5%Total costs 8,060,263 8,060,263Total benefits 129,229,894 208,173,452Cost/Benefit ratio 0.06 0.04Benefit/Cost ratio 16.0 25.8

Source: own estimations Importantly, one has to be cautious in interpreting these results. The implicit assumption here is that in a counterfactual scenario that is in absence of the Posgrado scholarship, none of the beneficiaries would have enrolled in postgraduate education. This assumption is not likely to be realistic given the nature of the program and the highly educated profile of beneficiaries. Unfortunately, there is no experimental or quasi experimental evidence that allow us to estimate the impact of this particular

41 The coefficient for postgraduate return is statistically significant at the 5 percent level and is statistically and significantly different from the educational level ‘completed university’. 42 We make a conservtive assumption and assume that workers will retire at age 60, the minimum age for retirement in Uruguay.

39

program in the probability of being enrolled into post-graduate education. To account for this limitation, two alternative scenarios are considered: (1) the program increases enrollment in post-graduate education by 50%43; (2) the program increases enrollment in post-graduate education by 20%. The IRRs estimated for these scenarios are 40% and 21% respectively. Table 3.5 and 3.6 present the cost-benefit results for the two scenarios. The Benefit/Cost ratio is positive even in the most conservative scenario.


Source: own estimations.


Source: own estimations. Iniciación a la Investigación scholarships

10. The evaluation of this activity conducted by the ANII (2012) shows that, among the undergraduate students who received an Iniciación scholarship in 2008, 73 percent had completed the bachelor’s program as of 2011 (versus 60 percent of the non beneficiaries), 62 percent obtained a research-related employment, and 34 percent of those who were employed before the scholarships declared that their wage increased as a result of the scholarship-financed outcome. Between 2009 and 2011, the treated group (scholarship beneficiaries) obtained on average 0.23 publication more than the control group, and this effect is attributable to the scholarship. An increased number of publications is well-known in sociology of science as to favor a "Matthew effect", a sort of multiplier effect for which the higher the number of scientific publications, the easier will be to publish in scientific journals in the future, the more profitable the career of the scientist (Merton 1968).

Component 3

Estimation of the Social Returns to Alliances

Main Assumptions

43 This scenario is in line with the impact found by Solis (2013), who estimates the effects of loosing budget constraint by providing accesss to credit loans on the probability of being enrolled in undergraduate college in Chile. Unfortunately, at the best of our knowledge there is no paper in the region that provides such estimate of the impact at postgraduate level.

40

i. Due to the lack of information on the distribution of pre-2012 investment by single year, pre-2012 investment is allocated to each year on the basis on the number of Alliances financed in the same time period.

ii. Economic benefits are assumed to materialize within one year since the beginning of participation in the program.

iii. The period 2010-2016 is considered.

Methodology

11. The following steps were considered to estimate the potential income estimations for firms that are in the Alliances program:

(a) Revenues before and after the beginning of the program were considered, for beneficiary and nonbeneficiary firms44;

(b) This difference-in-differences estimator (a naïve and non-conditioned one) was used as a proxy of the effect of the Alliances program and applied to the cumulative number of firms from the year they begin to participate in Alliances onwards.

Results

The IRR for the program is estimated to be 38%. Table 3.6 shows how the Benefit/Cost ratio is estimated to be greater than one45.

Table 3.6. Estimation of the Benefit/Cost ratio Discount rate 12% 5% Total costs 2,162,935 2,162,935 Total benefits 5,715,559 6,398,668 Cost/Benefit ratio 0.38 0.34 Benefit/Cost ratio 2.6 3.0

Source: own estimations.

Potential impact of instruments financed under Component 3

12. Aboal and Garda (2015) provide estimate of the production functions for private investment in innovation and innovation outcome respectively, for the case of Uruguay. These equations include a coefficient associated with cooperation of firms with at least one research institution, in innovation and research activities, which is useful to understand the potential impact of the instruments under Component 3 - public-private partnerships for innovation as Alliances, Networks and Centers.

13. Public-private cooperation for innovation in Uruguay is estimated to increase firms' investment in innovation by 52 percent compared to firms that do not participate in such

44 The time of implementation of the program varies for each treated firm, according to the year they begin to participate in Alliances. 45 The discount rate is applied only to benefits in one year (2016), given that costs and benefits for the previous years were already realized.

41

collaboration. Technological innovation output is 77 percent higher among participating firms and non-technological innovation is 400 percent higher.

14. Importantly, Networks and Centers imply not only collaboration between one firm and research institutions, but also cooperation with other firms. Crespi and Zuniga (2012) estimate that, for the case of Uruguay, R&D cooperation between firms could increase intensity of investment in innovation per employee by 0.53 percentage points. This positive result is higher than that obtained in Colombia (0.23 percentage points) but lower than the one estimated for Panama (1.34 percentage points).

42

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team Members

Name Title Unit Responsibility/

Specialty Lending Micky O. Ananth Program Assistant GTCDR Antonio Leonardo Blasco Sr Financial Management Specialist GGODR Maria Lucy Giraldo Consultant GGODR Pilar Elisa Gonzalez Rodriguez

Senior Counsel LEGCF

Jose L. Guasch Consultant GTIDR Thomas Edward Haven Senior Private Sector Development GTCDR Esperanza Lasagabaster Practice Manager GTCDR Matthew A. McMahon Consultant GFADR Natalia Moncada Senior Executive Assistant GSURR Gustavo Castro F. Raposo Finance Analyst WFALS

Edgardo Rubianes Consultant LCSHE - HIS

Kristian Thorn Education Specialist LCSHE - HIS

Supervision/ICR Diego Ambasz Senior Operations Officer GEDDR Micky O. Ananth Program Assistant GTCDR Gaston Mariano Blanco Sr Social Protection Specialist GSPDR Cidalia Brocca Finance Analyst WFALN Alejandro Caballero Senior Education Specialist CMGCS Pilar Elisa Gonzalez Rodriguez

Senior Counsel LEGCF

Robert J. Hawkins Senior Education Specialist GEDDR Esperanza Lasagabaster Practice Manager GTCDR Ricardo Eduardo Lugea Senior Procurement Specialist GGODR Mariana Margarita Montiel Senior Counsel LEGLE Xiomara A. Morel Sr Financial Management Specialist GGODR Alejandro Roger Solanot Sr Financial Management Specialist GGODR

43

(b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only)

Nº of Staff Weeks US$ thousand

(Including Travel and Consultant Costs)Lending FY05 0.00 7.72 FY06 18.69 107.50 FY07 39.71 143.53

Total: 58.40 258.75Supervision/ICR FY08 12.01 77.35 FY09 23.17 78.57 FY10 25.97 101.25 FY11 9.76 33.62 FY12 11.63 39.86 FY13 22.97 83.33 FY14 33.23 113.44 FY15 35.63 119.05

Total: 174.37 646.47

44

Annex 5. Beneficiary Survey Results

N/A

45

Annex 6. Stakeholder Workshop Report and Results

N/A

46

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR

Background

1. The need for the Project emerged as the result of a new policy initiated in 2005 to support the research and innovation sector, which was continued by the new Government. Within this framework, the agreement was implemented with the goal of strengthening the country’s capacity to create, transfer, and adapt knowledge and technology, funding the following activities:

The institutional framework for STI Investments in human capital and high-quality research equipment Transfer of technology and innovations in the private sector Collaboration and intersectoral and international mobility of researchers among public

institutions and the productive sector

2. In June 2007, the International Bank of Reconstruction and Development (IBRD) of the World Bank Group and the Eastern Republic of Uruguay signed the Loan Agreement, in which the Bank contributed US$26 million and Uruguay contributed US$6.5 million. 3. At first, the agreement was implemented by the MEC and the National Development Corporation (CND) until the ANII began to function, having been created by the law in December 2006. This new institutional framework was designed to bring together all the STI loans that existed in the country. 4. Once the agency was created, specific agreements with the Bank were made over time, culminating with the transfer of management responsibilities to the new institution. 5. Thus, the following milestones were achieved during the implementation of the Project:

(a) Effective transfer of the FM of the Project to the ANII. The definitive transfer of administrative and financial responsibilities from the CND to the ANII was accomplished during June 2008.

(b) Amendment to the Loan Agreement. On October 29, 2008, an amendment to the Loan Agreement was completed with the goal of updating the agreement to reflect the operational reality following the transfer of financial management to the ANII. The amendment included (a) establishment of the ANII as the program implementation agency and (b) clarification of the categories of eligible expenditures within the subproject (donation) framework.

(c) Subsidiary Agreement. Upon transferring the administrative and financial responsibilities from the CND to the ANII, it was necessary to execute a Subsidiary Agreement between the MEF and the ANII. Therefore, an agreement between the ANII and the MEF was signed.

(d) New version of the Operations Manual. On April 23, 2009, the Bank approved the new version of the Project Operations Manual with validity up to March 31, 2009. This new version of the Operations Manual reflected the new role of the ANII as a fully operational entity.

(e) Second amendment to the Loan Agreement. The MEF, the ANII, and the Bank worked on an amendment to the Project that would allow the investment in the TIMBO portal to

47

be recognized, as well as to update the result indicators based on the accumulated implementation experience that the Project had gained. The creation of the TIMBO portal stands out for its universal online access to international scientific and technological literature as well as to patent databases. The portal began functioning on January 1, 2009. The Bank formally sent the documents associated with the amendment to the Uruguayan Government authorities for their consideration and signature. They were signed by the MEF and the ANII.

(f) Project M&E indicators established in the PAD. The ANII proposed to the Bank modification of indicators that did not accurately reflect the Project activities. These changes were included in the documents that accompanied the amendment. The periodic measurement of these indicators is considered by the ANII to be a key element in program supervision. Accordingly, the following indicators were updated: (a) the PDOs and (b) the implementation indicators. Additionally, the ANII presented the M&E Plan for the various activities that the agency is undertaking.

(g) Project procurement issues. The ANII proceeded with the hiring of the firm Ernst & Young to carry out the sample-based financial and procurement audits of the subprojects. The Project employed a procurement specialist who was in charge of this area and whose principal mission consisted of developing tools to improve the conditions and the levels of control with which procurement is carried out, train supervisors and beneficiaries, and create and update the valuable information available through the ANII web page and the information systems in use (including the SEPA).

(h) Project FM. The Intermediate Financial Reports (IFRs) for each semester were received by the Bank without comments or observations. The financial information was presented correctly and the funds shown as received from the Bank matched the disbursement records. Consequently, the reports were acceptable to the Bank. An agreement was reached with the ANII to update the implementation status of the Procurement Plan each semester in the SEPA.

(i) External audit process. The audits were performed by private audit firms approved by the Bank using internationally recognized audit standards. These reports were accepted by the Bank, given that they contained the information required for Project financial statements and the funds shown as received from the loan matched the Bank’s records.

Promoting Innovation to Enhance Competitiveness Innovation Project 6. The Project consists of three components:

(a) Support for STI policies and institutional strengthening (b) Strengthening and guidance for research (c) Promotion of innovation and technology transfer to the productive sector

Component 1: Support for STI Policies and Institutional Strengthening 7. This component is subdivided into three subcomponents:

(a) Support for the development of public policies for STI (b) Support for the ANII (c) Creation of and support for the STI Observatory

48

8. This component will help establish a coherent policy framework for STI, consolidating and improving existing policies and programs. Subcomponent 1.1: Support for the Development of Public Policies for STI 9. In this subcomponent, various consultancies were undertaken and workshops were organized; an analysis related to the formulation and updating of the National STI Strategy was performed; coordination was undertaken with the key actors in the National Innovation System for the consultation processes; the dissemination of the National STI Strategy was maintained; and seminars and workshops were developed. These seminars and workshops dealt with the various consultancies that had been contracted to determine the state of the art in several knowledge areas. All of these consultancies were inputs for the development of the PENCTI, which was approved in February 2010. Analysis of STI Topics 10. Within the framework of this component, studies of STI indicators and the production of bibliometric indicators were funded. 11. The goal of this activity is to develop and foment the development of analytical works that have strategic value for decision making with regard to STI management at the discretion of the management committee, in addition to strengthening analytical capabilities on these topics at the national level. Consultants were hired to analyze specific STI thematic areas. Also, researchers were provided with access to the statistical information created and administered by the ANII for the development of research projects related to these topics. Analysis and Studies of STI Indicators 12. The goal of this activity is to develop a broad dissemination strategy for the information (indicators, studies, and diagnostics) created about the STI system that would allow it to reach different types of users: political decision makers, researchers, information producers, and society as a whole. Some of the topics included were results of the ‘3rd Survey of Innovation Activities in Industry (2004–2006)’; results of the ‘1st Survey of Innovation Activities in Services (2004–-2006)’; results of the ‘4th Survey of Innovation Activities in Industry’; results of the ‘2nd Survey of Innovation Activities in Services’; results of the ‘1st Survey of Innovation Activities in Agriculture’; results of the ‘National Survey of Public Perceptions of Science, Technology, and Innovation (2008 and 2011)’; the ‘2009 Survey of Public Perceptions of STI’; ‘Bibliometric Indicators’; and the National Bulletin of Science, Technology, and Innovation Indicators, among others. 13. An Innovation Indicators Workshop was organized at the Inter-American and Ibero-American levels and hosted in Uruguay jointly with the RICYT. The goal was to bring together researchers, producers of indicators, Government officials, and international organizations interested in new developments in the field of innovation indicators, to provide a space for both reflection and general methodological discussion of these types of indicators, as well as their analysis and potential uses as inputs into the political process. The number of individuals who participated in the workshop was 50, from more than 12 countries. The ANII provided the logistical inputs for the workshop, which took place in September 2010.

49

14. Additionally, the CVUy system was developed, along with a system for online project evaluation: The online résumé system, known as CVUy, is fully operational. An Online Project Evaluation System was also developed, which considered, in a general way, the varying possibilities and necessities for the evaluation of requests for projects related to the different programs and tools that the ANII implements. Bibliometric Indicators 15. The goal of the producing bibliometric indicators is to measure the results of the scientific and technological activities developed by investigators that work in Uruguayan institutions based on their publications in refereed scientific journals. A study of the evolution of bibliometric indicators in Uruguay during recent years was performed. Subcomponent 1.2: Support for the ANII 16. The formation and operation of the agency was funded when it began to operate in 2007 and 2008. The organizational structure was consolidated as a result of the development of information systems for the establishment of continuous improvement processes for management and the operational optimization to meet internal and external clients’ needs. The policies related to the implementation of frontline information tools were strengthened so that these tools would aid in the achievement of goals related to the foundational processes on which management and operations are based within the agency. Management processes were standardized and documented and workflows were automated, enabling the establishment of parameters for the various tasks and their measurement in the future. 17. Moreover, the Project supported the training of experts from the MEC and the ANII, offering them a master’s program in STI management from the National University of General Sarmiento in Argentina. TIMBO Portal 18. In this component, the electronic publications portal (TIMBO) was also funded. 19. The TIMBO portal, developed by the ANII during 2008 and made available to the system beginning in January 2009, has the goal of providing Uruguayan researchers and technology workers with universal online access to international scientific and technological literature as well as to patent databases. It is the first tool of its type in Uruguay and consists of a Government project focused on interdisciplinary scientific communication, ensuring equal opportunity to access the latest data in different knowledge areas. The portal provides free access and full text for journals, reference databases, and other publications, all selected based on their academic quality. 20. To create the TIMBO Portal, the ANII reached agreements with the principal international publishers, including Elsevier, for access to the Freedom Collection (1,650 journals across all disciplines) and Scopus (references from more than 14,000 works); IEEE (more than 1.8 million articles; 163 journals; 823 annual conferences; and 2,000 engineering, electronics, and telecommunications standards); Springer (1,300 journals and 6,400 books), among others. Moreover,

50

in 2009, new agreements were reached with Ebsco and Dot Lib, which incorporated more than 260 specialized publications in health and the social sciences beginning January 1, 2010. 21. With respect to the creation and support of the STI Observatory, the political timeline exceeded the temporary nature of the Project. 22. The implementation of Component 1 laid the foundation for the operation of the agency, the consultancies necessary to create the PENCTI, and the construction of the TIMBO portal. In this sense, the implementation of the program can be considered successful as a result of having complied with its mission and supported the new institutional framework in the area of STI, which marks a clear before and after in the country. Component 2: Incorporation of Human Capital and High-quality Research Teams 23. This component is divided in three subcomponents:

(a) Promoting research excellence in science and technology in priority research areas (b) Increasing the stock of advanced human capital (c) Strengthening priority tertiary technical education programs

24. This component will strengthen Uruguay’s ability to develop new knowledge. Subcomponent 2.1: Promoting Research Excellence in Science and Technology in Priority Research Areas 25. In this first subcomponent, the following activities were funded: 26. Projects of excellence in priority areas. These projects seek to invest in the consolidation of research and development capabilities in priority areas. In addition, these capabilities should focus on solving, over the medium term, the productive and/or social problems that are most relevant to the country. Calls for projects were launched and directed to consolidated, multidisciplinary and, as far as possible, inter-institutional groups, which proposed the creation of R&D and innovation projects aimed at finding solutions for important limiting factors in priority sectors. 27. In 2009, the ANII began to operate a new methodology of R&D and innovation program development and management, using Sectoral Funds in priority areas. Accordingly, the Energy Sector Fund began operations, created by a Board resolution with the participation of the UTE (the state electricity firm) and the ANCAP (the state petroleum firm). In coordination with the Ministry of Public Health, a similar fund was launched in the health sector. Additionally, the Innovagro Sector Fund began operations, created within the framework of an agreement with the Farming Research Institute at the end of 2008.

Energy Sector Fund. The goal of this tool is to promote research, development, and innovation activities in strategic areas—as defined in each request for projects—in the energy sector, through the financing of projects. This tool has two modes of operation, one aimed at research groups and the other at firms.

Innovagro Sector Fund. The general goal of this tool is to promote solutions for current problems, as well as those that are identified through forward-looking studies of the

51

development opportunities for agro-industrial value chains, preferably export focused. The Innovagro Sector Fund financed projects through two modes of operation: research projects aimed at strategic areas defined in each public request for projects and innovation projects requested by the productive sector.

28. In addition, the Social Impact Project was requested. This type of project seeks to invest in the development of research that has as its goal, the solving of problems whose solutions are characterized by a high level of social impact, in other words, higher levels of social well-being for the citizens of the country. 29. Projects in strategic areas with limited development were also funded. In this case, the goal is to strengthen strategic knowledge areas that could contribute to the country’s productive development and modernization but which currently lack consolidation from an academic point of view. Within this project framework, the Health Sector Fund was financed. The goal of this tool is to support R&D projects in Public Health and Human Medicine whose results are applicable to the national reality. These projects are focused on some of the areas/problems defined in the PENCTI. Subcomponent 2.2: Increasing the Stock of Advanced Human Capital 30. The goal was to strengthen educational offerings and create human capital in the technical and scientific fields. To achieve this goal, the subcomponent consisted of the following tools:

Support for National Postgraduate Programs. The goal of this tool is to develop and strengthen national master’s and doctorate programs by designing programs, training professors, and improving learning infrastructure.

National Postgraduate Scholarships. The goal of this tool is to strengthen the country’s human resources by funding national postgraduate scholarships (master’s and doctorates) in areas that have been deemed strategic by the GMI.

Strengthening Tertiary Technical Education Programs. This tool’s goal was to create and strengthen tertiary technical education programs—nonuniversity—in areas deemed to be priorities that are driven by an institutional arrangement where public and/or private entities participate and that deal with an identified need in the productive and/or social sector.

31. Furthermore, the subcomponent seeks to support those programs with competency profiles that were clearly defined, developed, and monitored in consultation with private enterprise. 32. In the same way that through Component 1 an innovative institutional framework was supported at the country level, in Component 2, through the strengthening of tertiary technical education programs, sectors that had not been previously addressed with programs focused on the productive sector were supported. In addition, scholarships for national postgraduate studies in high demand were consolidated significantly, while research projects in priority sectors with high demand were also supported. Support for national postgraduate programs was also a new tool in the country and entailed the creating and strengthening new programs of study in diverse areas.

52

Component 3: Promoting Innovation and Technology Transfer to the Productive Sector 33. This component is divided into two subcomponents:

(a) Development of sectoral technology centers (b) Support for public-private research consortia

34. This component entered the implementation phase later than the others did, but it has been seminally important for leveraging the transfer of knowledge to the productive sector. One of the tools used to foment the transfer of technology and innovation in the productive sector was the formation of Alliances among productive sector actors and those in the research and development sector through a tool called Innovation Alliances. This tool was implemented from 2010. 35. With the goal of promoting connections between the public and private sectors, the Sectoral Technology Networks component was also implemented, which included various segments/actors from the National Innovation System (public and private actors) concentrating and strengthening research, development, and innovation capabilities in priority sectors. 36. From 2012, various activities were developed with the goal of implementing Component 3 of the program. 37. Within this framework, it was understood that new tools must be designed that would finance potential technological needs in the country’s priority sectors and that would contribute to technology transfer. 38. In this way, two tools were designed: ‘Stimulating Technological Demand’ and ‘Sectoral Technology Centers’. 39. The Stimulating Technological Demand tool promotes innovation and technology transfer to the productive sector and foments relationships and connections among the various actors within the National Innovation System, with the goal of promoting their dynamism and competitiveness, thereby increasing their productivity and technological performance. 40. The Sectoral Technology Centers tool aims to transfer technology to the productive sector through diverse activities that these centers can undertake. Each sectoral technology center is a legally incorporated nonprofit organization that is statutorily obligated to contribute to the improvement of enterprises’ competitiveness by creating technological knowledge, undertaking R&D and innovation activities, and developing their applications. The knowledge application function comprises, among other things, carrying out projects with enterprises, providing innovation support services, training, and dissemination through technology transfer activities. 41. The goal of this tool is to strengthen Uruguay’s ability to offer specialized technology services focused on the productive sector in established and emerging sectors (in accordance with the priorities expressed in the national strategy) by creating new Technology Centers or improving the capabilities of those that already exist. Each Technology Center’s activities should allow for the establishment of new capabilities at the country level or strengthen existing ones.

53

42. This final component was, without a doubt, the most innovative in terms of the tools designed. In addition, it holds the greatest promise for future development. In addition to the Innovation Alliances and the Sectoral Technology Networks, which were successful, the Sectoral Technology Centers that are in operation are destined to close the gap between academia and the productive sector, in this way becoming the portion of the loan from which the greatest returns are expected. 43. The ultimate goal of the Project was to support the Government’s high-level goal of improving the Uruguayan economy’s competitiveness and stimulating growth by adding value to output and diversifying exports. 44. Specifically, the Project’s development goal was to strengthen Uruguay’s ability to create, transfer, and adapt knowledge and technology. In this sense, it can be confirmed that the Project is satisfying its development goal, but as has been seen in the experiences of the OECD countries, changing an innovation system is a gradual process that requires time and needs prolonged intervention. 45. The current Project is an important milestone in this journey and has demonstrated that the activities implemented have been successful in terms of achieving the development goal. Restructuring and Extensions 46. The original loan deadline was December 15, 2012. In this instance, a restructuring and extension of 18 months was requested, with the closing date moving to the end of June 2014. Subsequently, mainly based on the progress in Component 3, another extension was requested, moving the closing date to June 2015. 47. On February 3, 2014, it was requested that the MEF present the request for the extension of the loan implementation period to the Bank under the following terms: 48. “In 2013, in order to increase the number of projects performing technology transfer to the productive sector, projects were approved under the framework of a new tool, sectoral technology centers, and with extension, these projects can be fully implemented. Additionally, the extension will enable the research groups to complete their high quality studies and increase the amount of human capital funded by this program.” 49. In turn, the MEF presented the request to the Bank on February 5, 2014, supporting the ANII’s proposal and expressing that the extension was being requested for the implementation of the Sectoral Technology Centers and that an analysis of the extension had been undertaken with the Bank and the ministry during the Portfolio Review in November 2013. 50. This action extended the loan implementation period to June 12, 2015, at which point the financial execution of the loan was completed.

54

Intermediate Indicator under the Restructuring Framework 51. Within the framework of the restructuring, it was proposed that the ‘Average time between the request for proposals for subprojects and their final selection’ indicator be corrected. 52. The design of this indicator did not allow for the measurement of process efficiency in the assignment of subprojects. On the contrary, the indicator created perverse incentives to limit the time designated for the request for proposals. As a result, the ANII proposed considering modifications to this indicator as part of the restructuring process. 53. The new indicator was reformulated as the ‘average time between the close of the request for proposals for subprojects and their final selection’. Recommendations to the Bank and Lessons Learned 54. The Bank joined in the first phase of the creation of the ANII with support that turned out to be very important. Both in the loan preparation stage as well as at the beginning of the loan’s implementation the agency received ongoing advice from the Bank, which was reflected in the rapid start of the implementation phase. 55. With respect to the procurement procedures, work was performed in constant coordination with the Bank’s team to adjust the procedures to the Project’s reality, achieving the operation of a fluid contracting mechanism and processes that did not receive objections. 56. It is important to highlight the significant role that the Bank played in coordinating the visits to sectoral technology centers in Spain and Denmark before defining how the concept would be applied to the country’s reality. 57. As a general recommendation, it is proposed that an analysis of comparable experiences be performed for each loan component during the early stages of their implementation. 58. The institution evaluates the borrower’s performance during the preparation and implementation of the operation as very good. 59. The lessons learned from the Project’s preparation comprise the following:

Carry out prior research on the specific situation of each sector of the economy Analysis of the comparative experience of countries in the region to see how they can be

applied in the country Coordination with other multilateral lending organizations in terms of tools to be

implemented

60. With respect to the implementation period, the lessons learned comprise the following: Prior analysis of the demand for request for tools Support the rise of demand Study of the level of maturity of the innovation system to launch specific requests for

proposals

55

Measures Proposed for Future Operations 61. For future operations, it will be important to undertake an analysis of the strategies included in the five-year budget of each of the public ministries and institutions associated with the innovation-strengthening strategies at the country level. 62. In addition, it would be interesting to continue funding the Sectoral Technology Centers and monitoring their progress in transferring knowledge to the productive sector.

56

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders

N/A

57

Annex 9. List of Supporting Documents

Aboal, D. and P. Garda. 2015. Technological and non-technological innovation and productivity in services vis-à-vis manufacturing sectors. Economics of Innovation and New Technology.

ANII. 2012. Impacto De Becas De Iniciación A La Investigación. Documento de Trabajo n°3.

———. 2014. Informe de Evaluación Ex-Post, Becas de Posgrados Nacionales. Documento de Trabajo n°8.

———. 2015. Encuesta De Actividades De Innovación En La Industria Manufacturera Y Servicios Seleccionados (2010 -2012). Colección Indicadores Y Estudios Nr.9.

Bollen, J., Van de Sompel, H., Smith, J. and Luce, R. 2005. Toward alternative metrics of journal impact: a comparison of download and citation data. Information Processing and Management, Vol. 41, No. 6, pp. 1419-1440.

Bollen, J., Van de Sompel, H. and Rodriguez, M.A. 2008. Towards usage-based impact metrics: First results from the MESUR project. In Joint Conference on Digital Libraries (JCDL2006), Pittsburgh, PA, June 2008.

Bollen, J., Van de Sompel, H., Hagberg, A, and Chute, R. 2009. A principal component analysis of 39 scientific impact measures. PLOS ONE, Vol. 4, No. 6: e6022. doi:10.1371/journal.pone.0006022.

European Commission. 2007. Improving knowledge transfer between research institutions and industry across Europe.

———. 2012. Research infrastructures for the assessment of science, technology and innovation policy.

Gonzalez-Sabater, J. 2009. Manual de transferencia de tecnología y conocimiento.

Merton, R. K. 1968. The Matthew Effect in Science. Science

National Academies. 2014. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy .

Oddone (2011). Componente: Precios de Cuenta. Sub-componente: Tasa Social de Descuento. Fortalecimiento del Sistema Nacional de Inversión Pública. Convenio Oficina de Planeamiento y Presupuesto (OPP) y Facultad de Ciencias Económicas y de Administración (UdelaR).

OECD. 2005. The Measurement Of Scientific And Technological Activities - Oslo Manual.

OECD, Main Science and Technology Indicators.

Presidency of Uruguay. 2015a. Proyecto de Ley Sistema Nacional de Competitividad.

———. 2015b. Presupuesto Nacional 2015-2019.

58

Shepherd, P.T. 2007. The feasibility of developing and implementing journal usage factors: a research project sponsored by UKSG. The Journal for the Serials Community, Vol. 20, No. 2, pp. 117-123.

Solis, A. 2013. Credit Access and College Enrollment. The American Economic Association.

U.S. Department of Energy. 2011. Technology Readiness Assessment Guide.

World Bank. 2005a. Uruguay: Sources of Growth, Policies for the Development of Human Capital, Integration, Competition and Innovation. Report No. 31737-UY.

———. 2005. Uruguay: Country Assistance Strategy 2005–2010. Report No. 31804-UY.

———. 2007a. Uruguay: Promoting Innovation to Enhance Competitiveness. Project Appraisal Document.

———. 2007b. Uruguay: Promoting Innovation to Enhance Competitiveness. Loan Agreement.

———. 2008a. Restructuring Paper on a Proposed Project Restructuring of UY Promoting Innovation to Enhance Competitiveness Project.

———. 2008b. Restructuring Paper on a Proposed Project Restructuring of UY Promoting Innovation to Enhance Competitiveness Project.

———. 2010. Restructuring Paper on a Proposed Project Restructuring of UY Promoting Innovation to Enhance Competitiveness Project.



———. 2015. Uruguay Country Partnership Framework 2015–2020. Report Nº 97063-UY.

59

Annex 10. The Concepts of Transfer and Adaptation of Knowledge and Technology

1. A review of the sectoral literature (OECD 2005; European Community 2007; Sabater 2009; U.S. Department of Energy 2011) and of the international definitions and concepts (Oslo Manual 1997; OECD 2015; UNCTAD; CII) revealed that, although the two concepts are different (as specified in footnote 19), they are often used to refer to the same type of innovation activity but from different angles.

2. Transfer mainly refers to the knowledge and technology flows from the science and technology system to the productive sector, although it may also cover the transfer from the productive sector to other companies and markets, while adaptation is part of the adoption process for which the companies transform the created or transferred knowledge and technology into innovations in their specific contexts. For example, the Oslo Manual (1997) defines technological process innovation as the adoption of technologically new or significantly improved production methods, including methods of product delivery. Therefore, the adoption is at the core of the innovation activity, the output in which a certain transfer from science and technology can be an input (for example, through R&D, IPR agreements, mobility of expert technologists or scientists, linkages with universities or other innovative firms, and so on). These inputs can be considered investments for the firms (UNCTAD definition of transfer of technology, 2001). The adoption may also generate transfers to the rest of the world as part of the knowledge and technology diffusion. Therefore, transfer can be an input (the transferred or absorbed knowledge and technology from the STI system) and an output related to the diffusion of the adopted/adapted knowledge and technology. The concepts of transfer and adoption/adaptation are complementary, being the focus of the ‘transfer’ on the ‘supplier’ and the flows from STI suppliers to recipients. In ‘adoption/adaptation’, the focus is more on the internal implementation of the innovations by the final ‘user’, in particular, companies.

3. In practice, the different concepts are interconnected and not so easy to separate in statistical-indicators terms. The widest and more reputed statistical innovation data source is the EU Community Innovation Survey, exported to many developing countries in the world, also in Uruguay. In this survey, the adoption/adaptation is part of the innovation concept, and the transfer mainly comes under the innovation expenditure where firms declare their main inputs for their innovation. In the GII, the transfer concept is split between knowledge absorption (the transfer arriving to firms from the STI system) and knowledge diffusion with a set of symmetric indicators (for example, payments for royalties and licenses versus fee receipts for the same concept, high-tech import less reimports versus high-tech expert less reexport, or FDI net inflows versus FDI net outflows) that are useful at country level but not that much at the micro-firm level needed for an investment project. In the existing innovation survey and data sources, it is important to notice that the concept of ‘transfer’ is rarely used since firms often have difficulties in identifying the different types and sources of transfer and measuring them. As a consequence, there is no internationally established set of transfer and adaptation indicators suitable to be supplied in a separate and comparable way. In short, the concepts of transfer and adaptation indicators separated from the innovation concept in itself, since, in the case of the adaptation, it is part of the innovation concept in international definitions, and, in the case of the transfer, there are many and different heterogeneous indicators but reliable data at firm level are not so easy to obtain.

60

MAP