Serbia - Innovation Serbia Project - Implementation...
Click here to load reader
Serbia - Innovation Serbia Project - Implementation ...documents1.worldbank.org/curated/ru/394131476361888907/P... · Web view2016/10/08 · IMPLEMENTATION COMPLETION AND RESULTS
Serbia - Innovation Serbia Project - Implementation Completion and
Results Report(TF071742)
(US$11.9 MILLION EQUIVALENT)
CURRENCY EQUIVALENTS
Currency Unit = Euro (€)
EU IPA
FINS
FM
ICT
IMGGE
IMPR
IP
MOSTD
MOESTD
MPI
TA
Senior Global Practice Director:
E. Bank Staff………………………………………………………………………….ii
G. Ratings of Project Performance in
ISRs………………………………………......vi
H. Restructuring ……………………………………………………………………..vi
3. Assessment of Outcomes 16
4. Assessment of Risk to Development Outcome 27
5. Assessment of Bank and Recipient Performance 28
6. Lessons Learned 30
Annex 1. Project Costs and Financing 32
Annex 2. Outputs by Component 33
Annex 3. Economic and Financial Analysis 40
Annex 4. Bank Lending and Implementation Support/Supervision
Processes 41
Annex 5. Beneficiary Survey Results 43
Annex 6. Stakeholder Workshop Report and Results 47
Annex 7. Summary of Recipient's ICR and/or Comments on Draft ICR
48
Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders
50
Annex 9. List of Supporting Documents 51
MAP …………………………………………………………………………………..52
US$9.52 million
Disbursed Amount:
US$8.35 million[footnoteRef:2] [2: This system generated datasheet
only reflects Recipient Executed disbursement amounts and does not
reflect Bank Executed components or fees associated with this
hybrid trust fund. This table would suggest significant undisbursed
and cancelled resources at the end. However, only Euro 655,000 were
cancelled at project closing. Differences between the Revised
amount and the Disbursed Amount are most likely explained by the
Euro vs. US$ exchange rate differential between appraisal and
closing.]
Revised Amount:
US$7.15 million
Implementing Agencies: Ministry of Education, Science and
Technological Development (MOESTD); Serbia Innovation Fund
(IF)
Co-financiers and Other External Partners: The Donor is the EU
Delegation in Serbia
B. Key Dates
Outcomes:
Satisfactory
C.2 Detailed Ratings of Bank and Recipient Performance (by
ICR)
Bank
Ratings
Recipient
Ratings
Implementation Performance
No
No
Satisfactory
Original
Actual
Public administration - Financial Sector
10
10
Technology diffusion
Project Development Objectives (from Project Appraisal
Document)
The development objective of the project is to assist in building
the institutional capacity to stimulate innovative activities in
the enterprise sector by:
(a) Supporting the operationalization of the Serbia Innovation Fund
(IF);
(b) Piloting financial instruments for technological development
and innovation in enterprises; and
(c) Encouraging selected research and development institutes (RDIs)
to engage in technology transfer and commercialization, and
assisting in formulating RDI sector reform policy.
Revised Project Development Objectives (No changes, as approved by
original approving authority
(b) Intermediate Outcome Indicator(s)
Formally Revised Target Values
Indicator 1 :
Value
(quantitative
Number of IF managers trained and applying newly learnt
skills
Value
(quantitative
Value
(quantitative
Number of networking and educational events organized for
enterprises, academia and research
Value
(quantitative
Number of RDIs identified for technology transfer and
commercialization support
Value
(quantitative
Number of RDIs assessed jointly by international experts & RDI
management
Value
(quantitative
Indicator
Formally Revised Target Values
Indicator 1 :
Amount of innovation financing mobilized by IF in addition to this
EU IPA project
Value
Date achieved
Value
Number of new products and processes launched by beneficiary
enterprises
Value
Value
Value
Reason for Restructuring & Key Changes Made
DO
IP
No
S
S
6.36
Closing date extended from November 30, 2014 to January 10, 2016.
The extension allowed IF to complete core program activities,
including contracting of the new awardees from the fourth call for
proposals as well as disbursing earlier approved funds to the
existing awardees.
I. Disbursement Profile[footnoteRef:3] [3: This table would suggest
significant undisbursed and cancelled resources at the end. In
fact, there was a small cancellation of remaining funds at Closing,
equaling Euro 655,000. This chart is explained by the exchange rate
differential at appraisal and closing of the Euro vs. US$.]
3
1. Project Context, Development Objectives and
Design[footnoteRef:4] [4: This section is based on the Project
Appraisal Document (PAD).]
1.1 Context at Appraisal
1. The national innovation system and composition of research and
development (R&D) funding in 2012 did not support enterprise
innovation at any significant level, and the research sector
modernization agenda was in its infancy. The Serbian R&D sector
was dominated by the public sector, largely inefficient, and most
importantly delinked from industry needs, particularly those of
local small and medium enterprises (SMEs). The Serbian industry’s
capacity, both in terms of human capital and financial resources
for R&D and innovation, had been severely weakened
post-transition and exacerbated by the lingering effects of the
financial crisis. Brain drain was a major concern, in the private
sector and even more so in the scientific community.
2. Serbia spent roughly €100 million (0.3 percent of gross domestic
product [GDP] in 2010) on R&D, with a goal to reach 1 percent
by 2014. While R&D spending had increased from €28 million in
2001, Serbia still lagged significantly behind its neighbors.
Slovenia, the Czech Republic, and Croatia spend more than 1 percent
of GDP, and the European average was 1.8 percent. In Serbia, the
majority of R&D spending went to basic research, which
accounted for 50 percent of all R&D funding. A significant
shift from basic to applied research was needed for research
commercialization and development of an innovative SME sector.
This, along with developing stronger connections between science
and industry, was to be key in raising R&D expenditures to 2
percent of GDP by 2020, with a target of leveraging half of this
from the private sector.
3. Outputs from the R&D sector were not commensurate with the
public resources being invested and did not support modernization
of the Serbian economy. While the number of scientific publications
had increased during the previous few years, quality remained poor.
Intellectual property (IP) was either not being created or not
being protected. Merely 21 patents were registered by public
research and development institutes (RDIs) from 2003 to 2008, with
only 36 patent applications, mostly in Serbia. Figures for the
private sector were similar.
4. In an attempt to modernize the national innovation system, the
authorities had implemented a reform program over the previous two
years to revamp the regulatory and institutional framework
governing science, technology, and innovation. It had enacted a
number of laws to promote science and innovation and formulated the
Serbian Scientific and Technological Development Strategy
2009–2014. In addition, the Ministry of Education and Science was
created as the central body responsible for science and innovation
in Serbia.[footnoteRef:5] [5: The March 2011 restructuring of the
Government of Serbia (GoS) led to a merger of the Ministry of
Science and Technological Development (MOSTD) with the Ministry of
Education, thus creating a new entity, the Ministry of Education,
Science and Technological Development (MOESTD). The innovation and
technology development priorities have remained unchanged as a
result of restructuring. The review of programs described herein
was prepared by MOSTD.]
5. Science infrastructure improvement had been the main focus to
that point, while stimulation of private sector-led R&D and
innovation had been negligible. The European Investment Bank
provided a loan of €200 million to upgrade infrastructure at
universities and RDIs and to set up science parks as well as
Centers of Excellence in priority areas including biotechnology,
nanotechnology, and advanced computing. In 2010, the Government
disbursed approximately €70 million for basic research,
technological development, integral and interdisciplinary research,
and innovation activities. However, almost 80 percent of the funds
intended for science projects actually went to salaries for
researchers.
6. RDIs were the primary recipients of financing, with private
firms receiving funding only under the technological development
component. The 471 technological development projects supported by
the MOSTD were led by RDIs and not by industry. Public RDIs
depended on these projects as they did not receive regular
institutional budget allocations. Projects generally had a 90
percent approval rate, creating disincentives for researchers to
innovate. Except in a few cases, technology transfer and diffusion
from RDIs was quite low with only a few spin-offs and real
technology-based start-ups, indicating the need to place greater
emphasis on supporting the various stages of technology development
and innovation within enterprises—for example, technology transfer,
commercialization, diffusion, absorption, adaption, and
application.
7. Few mechanisms to incentivize private sector R&D and
innovation had been implemented to that point in time. The Ministry
of Economy and Regional Development (MOERD) administered the
Program to Strengthen Innovation in SMEs, that is, matching grants,
to support SME investment in innovation through co-financing.
However, the total budget for 2010 was RSD 40 million (€380,000).
All legal entities were eligible for co-financing of up to 50
percent of justified expenses for innovation activity. Matching
funds had to be provided from the SME’s own resources and could not
include any other public support. The awarded amounts could not
exceed RSD 800,000 (€7,500) for the first group of eligible
activities, and RSD 1.5 million (€14,000) for the second group. In
addition, under the Instrument for Pre-Accession (IPA) 2010
program, €3 million was allocated for the Integrated Innovation
Support Program with the objective to ‘increase the competitiveness
and economic growth in Serbia, through strengthening of innovation
in SMEs in accordance to National Strategy for Development of
Competitive and Innovative SMEs 2008–2013.’ However, implementation
of this program had not yet started.
8. In the post-crisis environment, the Government was expected to
play a critical role in stimulating economic recovery through
policies that target firm-based R&D and innovation. While the
Government and other international donors had demonstrated efforts
to support R&D and innovation, these were focused on renewing
infrastructure investments in public RDIs with negligible support
for private sector innovation. Given the World Bank’s experience
garnered in supporting private sector development in
post-transition economies, including in neighboring Croatia, the
GoS was keen on a World Bank engagement on the innovation agenda.
Hence, the World Bank’s technical assistance (TA) to the GoS and to
the future Innovation Fund (IF), being established under the
project as the implementing agency, focused on filling these gaps
and developing mechanisms to stimulate enterprise-led technology
development and innovation.
9. The objective of the FY12-15 CPS was to support Serbia’s EU
accession and help the Government strengthen competitiveness and
improve the efficiency and outcomes of social spending. Under the
Strengthening Competitiveness pillar Improved innovation capacity
was identified as one of objectives. The CPS noted that the system
and composition of R&D funding did not support Serbia’s agenda
to modernize and enhance its competitiveness. The CPS sought to
change this situation by supporting the establishment of
institutional capacity to stimulate activities in the enterprise
sector. The capacity would arise from creating a Serbia Innovation
Fund, using financial instruments for innovation and technological
development in enterprises, and having R&D institutes engaged
in transferring and commercializing technology and in helping
formulate RDI reform policy.
1.2 Original Project Development Objectives (PDO) and Key
Indicators
10. The project development objective (PDO) was to assist in
building the institutional capacity to stimulate innovative
activities in the enterprise sector by:
(a) Supporting the operationalization of the Serbia Innovation Fund
(IF);
(b) Piloting financial instruments for technological development
and innovation in enterprises; and
(c) Encouraging selected research and development institutes (RDIs)
to engage in technology transfer and commercialization, and
assisting in formulating RDI sector reform policy.
11. The key performance indicators for this project included:
(a) Amount of innovation financing mobilized in addition to this
European Union Instrument for Pre-Accession (EU IPA) project;
(b) Number of active technology start-ups funded;
(c) Number of new products and processes launched by beneficiary
enterprises;
(d) Number of technologies transferred by RDIs; and
(e) Dissemination of policy recommendations for RDI sector
reform.
1.3 Revised PDO and Key Indicators
12. The PDO was not revised. PDO-level indicators 1 and 2 were
refined for clarity’s sake during negotiations with the donor and
recipient, and are fundamentally the same as in the PAD.
1.4 Main Beneficiaries
13. The primary beneficiaries of this pilot project were the staff
of the IF whose capacity was built under Component 1;
entrepreneurs, start-ups, and firms that received financing and
mentoring/training under Component 2; researchers and leadership of
select RDIs that benefited from exposure to modern practices in IP
management, technology transfer, and commercialization, as well as
the Ministry of Education, Science and Technological Development
(MOESTD) that also benefited from the policy recommendations
developed based on the RDI TA program under Component 3. Important
secondary beneficiaries included stakeholders engaged during
project launch and implementation through numerous open
houses/workshops/conferences on such topics as entrepreneurship,
research commercialization, or innovation finance. These
stakeholders ranged from across Serbia’s national innovation
system, including leading universities, RDIs, incubators,
technology transfer offices, nascent early stage investors, and
private sector actors.
1.5 Original Components
14. In order to build institutional capacity to stimulate
innovative activities in the enterprise sector, the project design
intended to leverage three complementary components that would
enable piloting of institutional, financial, and commercialization
support mechanisms absent from the national innovation ecosystem
with the view to support and ideally demonstrate Serbia’s
enterprise innovation potential.
Component 1: Capacity Building of the Serbia Innovation Fund -
Recipient-Executed (€1.1 million)
15. The capacity of the IF was being established to encourage
entrepreneurship, including by financing enterprise innovation, as
well as participating in long-term programming efforts by national
authorities, international organizations, financial institutions,
and the private sector. In this context, this component provided
support for:
(a) operations of the Investment Committee and due diligence of
grant applications;
(b) engagement of strategic and operations advisors to assist the
IF’s current and future programs, strategy, operations, and
procedures;
(c) operational project management in relation to the European
Union (EU) and World Bank;
(d) IF staff training;
(g) IF operational infrastructure, including information and
communication technology (ICT) and financial management (FM)
development.
Component 2: Implementation of Financial Instruments Supporting
Enterprise Innovation - Recipient-Executed (€6.0 million)
16. This component financed (a) Mini Grants targeting the proof of
concept and prototyping stages, including but not limited to IP
protection and business plan preparation for initial capital
mobilization; and (b) Matching Grants targeting R&D in
technology development projects, for new or improved technologies,
products and processes. Incorporated entrepreneurs, innovative
start-ups, spin-offs, micro and small enterprises, with majority
Serbian private sector ownership were eligible for Mini and
Matching grants. The Mini and Matching Grants applications were
evaluated by the IF’s independent Investment Committee (IC), with
input from international peer reviewers.
17. The resources under this component were earmarked notionally
per instrument with the objective of flexible reallocation among
instruments based on demand. The terms and conditions of the grant
programs were designed with the IF advisors with sufficient
flexibility to accommodate changing market conditions and the
quality and volume of the project pipeline. A guiding principle was
to decrease the matching contribution from the IF over time,
particularly for the Matching Grants instrument.
Table 1. Design Features of the Mini Grant Programs at
Appraisal
Mini Grants
Baby Seed
€1,500,000–€3,000,000
Proof of concept, prototyping stage, IP protection, business plan
preparation for mobilization of initial capital
Recipient
Incorporated entrepreneurs, innovative startups, spin offs, micro
and SMEs, all with majority Serbian private sector ownership
Grant size
Up to €80,000 for projects that will be completed within 12
months
Features
Up to 85% of approved project costs
Table 2. Design Features of the Mini Grant Programs at
Appraisal
Matching Grants
€3,000,000–€4,500,000
Objective and stage
R&D (technology development) and commercialization projects for
new or improved technologies, products, and processes
Recipient
Incorporated entrepreneurs, innovative startups, spin offs, micro
and SMEs, all with majority Serbian private sector ownership
Grant size
Up to €300,000 for projects that will be completed within 2
years.
Features
Up to 75% Matching Grant with an optional 3–6% royalty component
based on sales revenue, up to 120% of the original grant within a
predetermined time frame
Component 3: Provision of Technical Assistance to Research and
Development Institutes (RDIs) - Bank-Executed (€0.7 million)
18. Component 3 was to provide (a) customized TA to up to two RDIs
based on a detailed needs assessment; and (b) limited TA to up to
four RDIs based on a general needs assessment; and (c) technical
input to the Government’s future RDI sector reform program based on
lessons learned from the aforementioned TA program.
19. A customized TA program for up to two select RDIs entailed
conducting a detailed assessment to be carried out by international
experts in partnership with the RDIs’ management. The assessment
was to cover issues, including but not limited to, the
organization’s management structure, human and budgetary resources,
researcher promotion and incentive system, infrastructure and
laboratory facilities, research capabilities and outputs, IP
protection, challenges and potential for applied R&D, and
technology transfer and commercialization. Based on this diagnosis,
technical support was to be provided for the design and
implementation of RDI-specific upgrading programs. Upgrading of
labs and other infrastructure would not be supported. The RDIs were
identified jointly by MOESTD and the World Bank based on several
key factors, including the level of commitment to reform from the
RDI management team, potential for improvement, likelihood of
success, sectoral coverage, and availability of resources. The
lessons derived from this program were expected to inform future
strategic planning and reforms in the RDI sector. Specifically, the
recommendations were expected to be focused and actionable, while
identifying quick wins. The World Bank team was to organize a
workshop with the ministry in charge to disseminate the experience,
lessons learned, and recommendations for a future RDI system reform
program.
1.6 Revised Components
1.7 Other Significant Changes
21. There were no significant changes. A simple project
restructuring was conducted with approval from the management of
the Europe and Central Asia Region and the donor to extend the
closing date from November 30, 2014 to January 10, 2016. The
extension allowed the IF to complete core activities, such as
contracting of awardees from the fourth call and disbursing
previously approved funds to awardees.
22. The US$ appreciated almost 30 percent against the EURO during
project implementation. The US dollar amount at project approval
was $11.9 million. The final disbursement amount – both Bank and
Recipient Executed - was approximately US$ 9.7 million, with
disbursement via the Recipient amounting to US$8.35 million. Of the
total EURO 8.4 million, EURO 655,000 was cancelled. Fortunately,
the exchange rate differential in dollars did not affect
implementation as the grant and expenditures were in Euros.
2. Key Factors Affecting Implementation and Outcomes
2.1 Project Preparation, Design and Quality at Entry
(a) Soundness of the Background Analysis
23. The background analysis for this project was sound, given its
pilot nature and little government experience in supporting
enterprise innovation or RDI sector reforms. The project was
aligned with the FY08–FY11 Country Partnership Strategy, wherein a
key priority included encouraging dynamic private sector-led growth
to promote convergence with European levels. In FY09 and FY10, the
World Bank team delivered a TA program, supporting MOESTD to update
legislative frameworks that encouraged firm uptake of innovation
support. Specifically, the World Bank team worked on identifying
challenges with existing innovation programs, reviewing the legal
framework governing R&D and innovation, and a new innovation
strategy. Further, at entry, the focus was on (a) appraising
institutional arrangements supporting research and innovation; (b)
surveying and assessing firm perception of existing innovation and
technology absorption support programs and appetite for proposed
instruments; and (c) discerning level of opposition for support to
firms and for research sector reform. The analysis and
consultations were instrumental in sharing regional and global
experience, building stakeholder confidence on the need to deploy a
pilot project that would initiate a steady transition to an
enterprise-led model.
(b) Assessment of the Project Design
24. Pilot. The project was designed carefully as a modest pilot
initiative that aimed at building the missing institutional
capacity in the Serbian innovation system, testing financial
instruments and mentoring mechanisms to reveal the emerging
innovative entrepreneurial sector, and identifying viable
approaches to the challenges of Serbia’s ailing research sector.
Given high expectations from this pilot, the risky nature of both
demand- and supply-side interventions, accompanied by significant
latent opposition from researchers in the public RDIs, the split
between World Bank and recipient-executed components was prudent.
Equally judicious was the attraction of top global expertise to
build credibility for this pilot, and demonstration of key
principles in innovation policy management, including
horizontality, independence, good governance, efficiency,
transparency, monitoring and evaluation (M&E), and open
communication with stakeholders.
25. Selection Procedures. Noteworthy design features included the
IF’s efficient and independent grant selection processes that
engaged international peer reviewers and an IC featuring
international and diaspora experts with a background in scientific
research, entrepreneurship, technology commercialization, market
trends, and venture investments.
26. Visibility. The donor placed significant emphasis on organizing
project visibility activities and these ended up being instrumental
given the risky and novel nature of the pilot. Most visibility
activities were high level in nature and co-hosted by the
recipient, the donor, and the World Bank, which helped reinforce
the Government’s commitment in spite of multiple ministerial
reshufflings within the lifetime of the project.
27. Ownership. The World Bank-executed TA component was designed to
support knowledge transfer and research commercialization within
the lifetime of the project. Only those RDIs where management
demonstrated (a) willingness to be assessed by international expert
teams convened for their institution and (b) commitment to act on
key recommendations from such assessments, were included in the RDI
TA program. This allowed the World Bank team to narrow down the
list of RDIs it chose to provide TA support. In retrospect, this
important design feature afforded flexibility, as both time and
financial resources were limited.
(c) Adequacy of Government’s Commitment
28. The Government’s commitment toward the project—and especially
for the enterprise innovation objectives—was high at entry. The
Government demonstrated this by pursuing the Delegation of the
European Union (EUD) to earmark EU IPA funding for enterprise
innovation for the very first time, approving a notional
operational budget for the IF from 2011 to 2015, and attracting a
senior diaspora member to lead the IF. The World Bank was nominated
by the GoS to enter into a trust fund arrangement and execute
Component 3, which was controversial at the time.
(d) Assessment of Risks
29. Several risks were identified during project preparation and
adequate risk mitigation measures were incorporated into the
project design as follows:
· A pioneer on several fronts, there were important coordination
and reputational risks stemming from this pilot. This was the first
ever EU-financed World Bank-administered trust fund in Serbia,
addressing unchartered policy territory, facing skepticism by
entrenched interests and significant political instability. Through
meticulous supervision and dialogue with the EUD, MOESTD, MOERD,
IF, and other stakeholders during missions and steering committee
meetings, the World Bank fostered shared awareness on project
progress and ownership of challenges as they arose.
· Significant resistance from entrenched interests to RDI sector
reforms. The World Bank team managed the RDI TA program closely,
expanding the scope of assistance only to volunteer RDIs where
management demonstrated firm commitment to optimization and
knowledge transfer and research commercialization outcomes.
· The IF had no demonstrable capacity to transparently select
projects, manage FM, procurement, environmental screenings, or
support the private sector. The World Bank’s no objection to the
selection of IC members, and to procedures followed during the
first call, supported the credibility of the selection processes.
The international advisors provided continuous capacity building
for IF staff on program design couched in global practice, whereas
the IC safeguarded the IF’s independence, together tempering undue
influence from the research community or political
interference.
2.2 Implementation
30. The following factors had an impact on project
implementation:
· Factors having a positive impact
· The establishment of the IF as an efficient independent
institution with strong and transparent governance practices was
instrumental in smooth project implementation.
· The independent and professional manner in which beneficiaries
were selected and monitored during implementation ensured that the
selected subprojects were successfully implemented.
· Continuous access to the IF Advisory team, oversight from the
World Bank, and access to MOESTD, EUD, and the IF’s Board, was
important in allowing the IF to overcome teething and political
challenges over the course of the pilot, focus on creating a
positive environment for its beneficiaries, and introduce new
innovation and technology transfer programs for Serbia.
· In line with the concept of the pilot, both recipient- and World
Bank-executed components were monitored closely and pivoted
incrementally to enhance program outcomes and inform longer-term
instrument design. A few examples include design of the mentoring
program to accommodate mandatory and customized training elements
for IF staff and beneficiary entrepreneurs; deploying the Mini and
Matching Grant programs sequentially so the IF could incorporate
lessons from the first call of the Mini Grant Program; increasing
the private sector contribution (from 25 percent to 30 percent )
under the Matching Grant program; converging on design details for
the royalty scheme; and engaging commercialization brokers to work
with RDIs to identify commercialization wins within the lifetime of
the project.
· Under Component 3, the growing acceptance of knowledge transfer
and research commercialization was an important cultural shift
among researchers and managers in the RDI community, enabling the
recipient to consider sector wide reforms in the future.
· Continuous access to the IF Advisory team, oversight from the
World Bank, and access to MOESTD, EUD, and the IF’s Board, was
important in allowing the IF to overcome teething and political
challenges over the course of the pilot, focus on creating a
positive environment for its beneficiaries, and introduce new
innovation and technology transfer programs for Serbia.
· Factors having a negative impact
· 2012–2013. During the early stages of project implementation,
disbursements were lower than planned at entry. This was due to GoS
insistence at entry to plan as many as two Mini and two Matching
Grant Calls for Proposals (CFPs) in 2012. The IF had just been
established and did not have the capacity at the time to handle the
proposed workload. However, based on guidance the IF received from
its advisors, it was eventually agreed to phase the rollout of the
programs and to pilot the first CFP for the Mini Grant Program
separately, and incorporate any lessons from this call into future
ones. In retrospect, this was an important adjustment, enabling IF
staff to be better prepared for subsequent calls.
· 2014. Serbia faced a turbulent year in 2014 and so did the IF.
This included catastrophic flooding, multiple changes in the
Government, delayed funding, and management restructuring within
the IF. Implementation progress slowed due to the lack of budget
funds for the IF’s operations and given the uncertainty, the IF
experienced significant staff departure. Additionally, MOESTD’s
interest in engaging the World Bank on the public research sector
reform agenda diminished in the absence of a larger World Bank
operation. The project was downgraded from Satisfactory to
Moderately Satisfactory. Nonetheless, with guidance from its
advisory team, the IF was able to rebuild staff capacity by
conducting additional trainings to bring the new cohort to speed.
During this period, project implementation continued, albeit at a
slower speed. By July 2014, the IF was fully staffed, the budget
fully funded, and operations commenced satisfactorily. After the
mid-term review in August 2014, the project was again rated
Satisfactory, and this status was maintained through closing in
January 10, 2016.
2.3 Monitoring and Evaluation (M&E) Design, Implementation and
Utilization
Rating: Substantial
31. M&E design. There was no systematic effort to capture
firm-level innovation or research commercialization data in Serbia
at entry. Publicly available indicators were limited to typical
inputs (for example, public R&D spend) or research outputs (for
example, publications and patents). Given minimal baseline
information, skepticism around the pilot in Serbia, the growing
debate on effectiveness of Matching Grants in the World Bank, and
high donor and recipient expectations, the M&E activities
envisaged under this project were deliberate. Beyond the set of
input, output, and outcome indicators in the results matrix, the
project focused on building the IF’s long-term capacity to identify
additional outcome indicators, build its internal systems to
capture and analyze firm-level innovation data to support future
evidence-based policy making in Serbia beyond this modest pilot. At
project inception, M&E activities were managed and supported by
two IF staff members, a program manager, and a senior associate,
whose tasks entailed assisting in data collection, overseeing
deliverables, and payment schedules. However, as implementation
progressed, the IF realized that it lacked the M&E expertise to
establish and operate an in–house system. As a result, a reputable
external think tank with a track record in conducting innovation
program impact evaluations was engaged by the IF from May 2013 to
November 2015 to ensure the quality of methodology and
implementation, and to design the most relevant approach for the
small sample size of beneficiaries expected under the pilot.
32. M&E implementation. The consulting firm developed a
framework and procedures for M&E activities under the project
which was incorporated in an M&E manual for the IF, and in
addition, trained the relevant staff. Staff training provided by
the external consultant focused on identifying relevant indicators,
monitoring processes, specifically deploying IF data collection
systems. As part of their assignment, the firm and IF gathered
extensive data on both beneficiaries and applicants. M&E
functions were to monitor and analyze the performance of the Mini
and Matching Grant programs’ subprojects. IF staff were also
trained to monitor financing programs to be managed by the IF in
the future. The IF produced semiannual M&E reports. These
reports were shared with the Government, World Bank, and EUD, and
served as a useful tool for project monitoring and progress
reporting. The independent evaluation was led by an independent
firm. For the World Bank-executed component, M&E data was
collected and analysed by the World Bank team.
33. M&E utilization. The IF and the World Bank monitored the
interim outputs of the project systematically to assess progress
toward end-of-project target outcomes, identifying issues and
risks, and reporting on time-bound action plans to mitigate them.
In August 2015, with participation of IF staff, the consulting firm
presented its findings to key stakeholders, including senior
representatives from MOESTD, MOERD, EUD, Ministry of Finance, and
IP office, among others. The focus of the presentation was on the
selection process, demonstrating the additionality of IF programs
and recommendations for scaling of future programing. The
evaluations of the Mini and Matching Grant programs justified the
EUD’s decision to provide additional funding for these programs
under its future programming and also justified inclusion of these
programs under the World Bank-funded Competitiveness and Jobs
Project (C&JP), a results-based operation (2015). On Component
3, the monitoring indicators proved useful in concentrating efforts
of RDI management and researchers on knowledge transfer and
commercialization activities. Importantly, it allowed the World
Bank team to estimate the financing gap for knowledge transfer and
research commercialization in Serbia, enabling the GoS to advocate
for the sequel EU IPA-financed program with the EUD, which
supported the establishment of the national Technology Transfer
Facility (TTF) and the Collaborative Grant Scheme under the ongoing
Serbia Research, Innovation and Technology Transfer Project
(SRITTP).
2.4 Safeguard and Fiduciary Compliance
34. The project did not entail any major environmental risk,
include construction or land acquisition. The project was rated
category ‘B’. There were no issues in compliance with environmental
safeguards. Due to the limited size and scope of the grants, there
was no significant environmental impact associated with the
project’s activities. The overall innovation process under this
component did not support environmentally unfriendly technologies
and practices. The Environmental Management Framework was
implemented successfully. The project did not entail any social
risk or risk of adverse social impact.
35. FM was assessed as Satisfactory. There were no outstanding
audit reports for the project and all audit reports were found
satisfactory to the World Bank. Quarterly interim unaudited
financial reports were submitted to the World Bank on time. The
project audits received ‘unqualified’ opinions of the audited
financial statements and no major systems and control issues were
identified by the auditors.
36. Procurement was rated Satisfactory, based on review of prior
review documents and post review missions. The IF and beneficiaries
followed the World Bank's procurement guidelines and no major
issues were encountered. For procurement under the grants, World
Bank-approved simplified commercial practices were used.
2.5 Post-completion Operation/Next Phase
37. In 2014, the Government was unable to finance post-pilot calls
for the IF grant programs as budget resources were diverted to
post-flood remediation efforts. However, key elements for a
post-pilot transition are now in place with the approval of the EU
IPA SRITTP and the C&JP operations in 2015. SRITTP provides
support for national technology transfer activities, collaborative
industry/academia research activities, and support for development
of the new strategy, the infrastructure road map, and their related
action plans. C&JP supports among other activities, the
integration of the IF’s operational budget into the GoS’s annual
budget, continuation of the Mini and Matching Grant Programs,
scaling of technology transfer initiatives, and RDI sector reforms
through a results-based approach.
3. Assessment of Outcomes
38. There were no major shortcomings in achievement of project
objectives, efficiency, or relevance.
3.1 Relevance of Objectives, Design, and Implementation
(a) Relevance of Objectives
39. The relevance of the objective is rated High. The objectives
and related outcomes of this project are as relevant at the time of
the Implementation Completion and Results Report (ICR) as they were
at entry and remain strongly aligned with the World Bank’s
engagement in Serbia and the Government’s development agenda. This
project remains highly relevant for the overarching objectives laid
out in the FY16-20 CPF which is to support Serbia in creating a
competitive and inclusive economy and, through this, to promote
integration into the EU. One of two broad CPF focus areas - Private
sector growth and economic inclusion should address significant
constraints to private sector development and economic inclusion:
financing, investment, connectivity, and labor market constraints.
This Project was both timely and relevant as it supported the
creation of institutional capacity and programming for private
sector innovation that can be scaled with future IPA and eventually
EU funding.
40. The GoS is undertaking measures to rebalance the economy,
enhance the quality and efficiency of public expenditures and
service delivery (including state-owned enterprise reform), and
stimulate private sector initiatives to enhance Serbia’s
competitiveness. In March 2016, the Government adopted the Strategy
for Science and Technology Development: Research for Innovation
2016–2020, specifically calling for reforms in the public RDI
sector and reinforcing the importance of enterprise innovation and
technology transfer for the economy, including support for the
activities carried out by the IF. This pilot furnishes evidence to
engage on innovation policy and is demonstrated by GoS engagement
in the SRITTP and C&JP.
(b) Relevance of Design
41. The relevance of the design is rated Substantial. Project
design was essential in supporting the delivery of PDO outcomes and
enabled successful piloting of missing elements of the Serbian
innovation system. The core design features remain relevant and are
therefore being continued or expanded under SRITTP and C&JP.
They would warrant being supplemented by increased research
infrastructure investments, coupled with reforms in the public
research sector, and direct support instruments for enterprise
innovation and market access in the future. Under Component 1, the
IF leveraged funding and partnerships with international donor
organizations to strategically build entrepreneurship and
innovation programming from traditional early-stage matching grant
funding (SRITTP, C&JP) to early stage equity funding (that is,
Western Balkans Enterprise Development and Innovation Facility [WB
EDIF]). Through Component 2, the IF demonstrated long-term demand
for early-stage enterprise innovation funding and the ability of
Serbia’s private sector to lead innovation projects (often in
partnership with the research sector) and co-finance these with own
matching resources. Under Component 3, the World Bank managed to
provide in-depth TA to four RDIs and deliver technology transfer
targets by being selective and staying focused. This built the case
to establish a national TTF to stimulate collaborative research
under the SRITTP and support RDI sector reform activities under the
C&JP. At entry, there were no visible private ecosystem players
present in Serbia; hence, the pilot could not be designed to engage
them. In future iterations of innovation programming and financial
instrument design (including of the Matching Grant programs), the
IF could explore leveraging local private ecosystem actors—for
example, business angels, other early stage investors,
accelerators, and so on—in its selection, co-funding, and/or
mentoring processes.
(c) Relevance of Implementation
42. Relevance of implementation is rated Substantial.
Implementation outcomes remain relevant, although implementation
arrangements can be further strengthened in the future. The
emphasis placed on good international practices, efficiency,
horizontality, independence, and transparency is deemed relevant
for the implementation outcomes of the pilot. So much so, that the
donors and the World Bank continue to leverage the IF and MOESTD
for additional innovation activities under the SRITTP and C&JP.
Although currently manageable, if the IF and MOESTD become solely
responsible for innovation, without commensurate staffing and
capacity building in the new areas of responsibility (for example,
technology transfer), there could be trade-offs in terms of
quality, delivery, and overall direction of the agenda. Further
expansion in the IF’s implementation responsibilities ought to be
accompanied by wider ownership and endorsement by key line
ministries of the cross-cutting research and innovation agenda,
such as ministries in charge of the finance, economy, agriculture,
and so on.
3.2 Achievement of Project Development Objectives
43. Achievement of PDO is rated High. Overall, the project achieved
or exceeded its stated objectives of building institutional
capacity to stimulate innovative activities in the enterprise
sector as explained below. This was achieved through the
operationalization of the IF, piloting of financial instruments for
technological development and innovation in enterprises; and
encouraging selected RDIs to engage in technology transfer and
commercialization, and assisting in formulating RDI sector reform
policy.
44. Operationalization of the IF. As the main implementing agency,
the IF was responsible for (a) successfully designing, piloting,
and managing enterprise innovation programs; (b) supporting proof
of concept and prototyping activities and demonstrating a strong
pipeline of entrepreneurs and innovative firms in Serbia; and (c)
participating in co-financing of programs and leveraging other
activities organized by international organizations, financial
institutions, and the private sector, all to further the innovation
agenda on behalf of the GoS. The IF engaged its international
strategic, operations, and regional advisors to develop its own
capacity and operational procedures and provide guidance to
enterprises and the GoS on enterprise innovation policy and
programs.
45. Building Institutional Capacity within the IF and Beyond. An
essential service provided by the IF was the mentoring provided
through Enterprise Training, in addition to the specialized
training programs that entrepreneurs would elect themselves. This
was conducted by international advisors covering key innovation
management issues, such as business formation and expansion, IP
protection and commercialization, investment readiness, and early
stage technology and business development. Entrepreneurs reported
these trainings to be very useful, valuable, and otherwise
inaccessible. This training also prepared IF staff for the advanced
capacity building led by the advisory team, and especially for the
international trainings in Israel, Finland, and Croatia that
focused on ecosystem development, institutional, policy,
early-stage entrepreneurship and innovation program design,
sequencing, and evaluation. The IF today is well-regarded by its
peers, and newer innovation agencies, including Macedonia and
Georgia, have requested the IF to conduct training programs for
their staff.
46. Supporting Evolution of the Ecosystem. The IF played a vital
awareness-building role for the nascent innovation ecosystem by
conducting around 22 educational and networking events for its
beneficiaries and a broad set of institutional stakeholders. These
activities included project launch activities across six cities in
Serbia, successive launches of the Mini and Matching Grants
Programs, IP and Technology Transfer Workshop, WB EDIF workshop and
four info-days for the awardees (addressing project implementation
and training held by the Serbian IP office), four Enterprise
Training Sessions for awardees, presentation of the IF Annual
Report, Finland-Israel Knowledge Economy Workshop, consultations
with four major technology transfer offices on the design of the
centralized TTF, industry-RDI consultations on the design of the
Collaborative Grant Scheme, and project closing.
47. Leveraging of Resources by the IF. During design, the intention
was for the IF be established as an entity that is both experienced
at managing and capable of raising additional funds beyond this
pilot. The aforementioned factors and IF’s strong delivery
positioned it as an institution with significant credibility to
manage the EU and other donor funds in a professional independent
and efficient manner, far beyond what was originally planned at the
onset of the pilot, including for the following activities:
(a) WB EDIF (€140 million). The IF led the preparation of this
regional program and mobilized €28 million from the GoS.
(b) SRITTP (€6.9 million). A sequel to the current pilot, the
SRITTP is financed through the 2013 EU IPA.
(c) Serbia C&JP (€89.5 million). Under C&JP, the IF can
access €12 million for its operational budget, financing of new
calls for the Mini and Matching Grant Programs, and technology
transfer activities.
44. Piloting financial instruments. The responsible governance
mechanisms established by the IF—through its 60-plus independent
international peer reviewers, 5-member professional diaspora-led
independent IC, efficient under-90-day selection procedures
codified in its Mini and Matching Grant Manuals (GMs)—underscored
the transparent merit-based selection processes it developed. The
IF completed four CFPs for the Mini and Matching Grants programs by
October 2013, announcing the last batch of awardees on February 14,
2014. Funding was awarded for 41 Mini Grants and 11 Matching Grants
to start-ups and firms with innovative projects and high potential
for commercialization. Nineteen new products and processes were
launched by the beneficiary companies, over 300 high-value jobs
were supported, and 51 companies were mentored through Enterprise
Training.[footnoteRef:6] By closing, €5.4 million had been
disbursed to awardees who expressed their gratitude to the IF and
authorities for designing and managing the grant facility in a
transparent manner. Several of these early-stage projects are
beginning to demonstrate successes, with some receiving noteworthy
coverage of their achievements in the local and international
press, including in the Financial Times, the BBC, and the New York
Times. At the time of ICR completion, 36 of the 38 startups funded
under the Mini Grant program and all 10 of the small and medium
size firms financed under the Matching Grant program were still
active, demonstrating a 95 percent survival rate. In line with
international good practice, this pilot project was subject to a
rigorous independent evaluation and annex 5 captures both key
findings on additionality of programs as well as critical
recommendations for future scale up by GoS. [6: Detailed
description of each sub-grant and the amounts disbursed can be
found in project files. Annex 2 provides a few examples of Mini and
Matching Grant Projects. ]
Box 1. Case Study of a Successful Mini Grant Program: Start-up
Strawberry Energy
Strawberry Energy was a Mini Grant recipient. This Serbian startup
made renewable energy sources more accessible by developing,
designing, and implementing clean technologies. It developed a
solar charger and then applied the expertise gained to create the
‘Strawberry Tree Mini’—a flexible and transportable mini solar
charger for cell phones—with IF financing. Besides improved
characteristics of power production, consumption management, and
functionality of solar panels and batteries, the mini charger
includes an interactive component that shares renewable energy
information with end users. A finalist of the 2013 World Technology
Award, Strawberry Energy has been recognized as an innovation
leader in environmental and energy efficiency. Strawberry Tree was
featured on Mashable among 25 top technologies every smart city
should have. Following completion of the IF project, Strawberry
Energy has continued development of its technology and has created
the ‘Strawberry Mini Rural’, a solar charger suitable for areas
with low access to electrical energy. The Accelerator Venture Fund
Eleven from Bulgaria made a €100,000 investment in Strawberry
Energy in 2014. In October 2014, Strawberry Energy was the only
participating company from outside the United States to win the
international start-up contest organized by the GreenBiz Group,
held in San Francisco, California.
Table 3. Additional Achievements of Mini and Matching Grant
Awardees
Activity
19
2
13
4
5
Source: IF.
45. Technology transfer and commercialization. Another activity
that this pilot tackled was to encourage RDIs to engage in
institutional assessments and knowledge transfer and research
commercialization activities to glean insights for sector wide
policy reforms. The World Bank team conducted detailed diagnostic
assessments of four RDIs (classified as Category I); that is, two
more RDIs than planned at entry: Institute of Physics, Belgrade
(IPB), Institute of Molecular Genetics and Genetic Engineering
(IMGGE), Institute of Food Technology (FINS), Novi Sad, and
Institute of Medicinal Plants Research (IMPR). These assessments
provided detailed recommendations (and specific action plans) for
improved institutional capacity, performance management, and
knowledge transfer and research commercialization practices that
were the basis for customized technical support provided to these
RDIs. As part of the TA, the World Bank organized trainings for a
broader group of 10 RDIs (classified as Category II) to identify
opportunities for institutional improvements. These institutions
participated in discussions to identify cross-cutting topics of
interest to RDIs, including: Program/Project Management, IP
Management, R&D Marketing and Sales, Performance Evaluation and
Career Planning/Development, and Managing/Developing R&D
Capabilities. Based on expressed interest, the World Bank delivered
three high-level training workshops on topics of interest to the
RDI sector community.
46. Research Sector Reform. Given its modest means, the TA program
made important contributions to the voluntary institutional
adjustments and technology transfer efforts at IPB and IMGGE. These
two RDIs were responsible for three of the early-stage technology
transfer targets met under this project. However, supporting
technology transfer and commercialization was very challenging
owing to basic research orientation of most RDIs and the weak
institutional capabilities, low level of technological and even
lower market readiness of the proposed R&D projects. With the
establishment of an Innovation Center at the IPB, technology
transfer activities progress picked up during the project. IPB
leadership as well as many department heads and researchers
demonstrated strong interest and willingness to undertake some
difficult institutional and mind set changes. The IMGGE transformed
itself from a university-like organization to a research institute
comparable with its international counterparts and at closing was
on the verge of exporting a new product supported under the TA (see
annex 3 for details). Both RDIs reported a significant change in
attitudes among its management teams and researchers in engaging
the private sector on potential knowledge transfer and
commercialization projects. Both RDIs reported the culture as more
favorable toward conducting applied research with commercialization
potential by partnering with the private sector, internal reforms
in performance and institutional management, and the need to look
for non-budgetary sources of revenues for the future. Both
RDIs were able to obtain several knowledge transfer contracts from
the private sector firms and European organizations, produce patent
filings and high quality publications.
47. The management of the FINS and the IMPR—the two other RDIs
supported under the detailed TA—appreciated the in-depth
assessments, but found it difficult to motivate staff to proceed
with the recommended institutional reforms without an explicit
mandate push or support from MOESTD.
48. Overall, this World Bank-led activity provided extensive
insight into the market relevance of the RDI projects and revealed
the financing gap to support commercialization and conduct RDI
sector reforms nationally. In October 2013, the World Bank
delivered a Policy Note on RDI Sector Reform to MOESTD. In 2015,
subsequent MOESTD leadership took up recommendations from the
Policy Note to initiate systematic reforms in the R&D sector.
The World Bank and IF teams also provided specific inputs to the
ministry officials for the design of two projects: (a) the EU
IPA-funded SRITTP and (b) World Bank’s C&JP. For instance,
under the C&JP, the GoS is adopting a strategy that commits to
undertaking reforms in the public research sector by conducting
RDI-level assessments with global experts and establishing
market-aligned key performance indicators that are for both
individual researchers and RDIs. Both projects were approved and
currently are under implementation.
3.3 Efficiency
Rating: Substantial
49. Overall, this pilot achieved its objectives efficiently and is
rated Substantial. It is not useful to conduct a cost-benefit
analysis for a project (a) that was attempting to pilot multiple
missing elements in an innovation ecosystem; (b) that was going to
trigger systemic changes; (c) that required significant scaling of
financing to demonstrate impact; and (d) where the full effect of
the outcomes would be seen only several years post pilot. Comparing
costs of similar programs elsewhere can lead to erroneous
inferences as initial circumstances vary by country. The PDO was
successfully achieved with the operationalization of the IF (€1.1
million) and the project achieved better outcomes than originally
envisaged: (a) leverage of at least €35 million through three donor
projects versus €20 million target; (b) grant decision making in
60–90 days versus regional average of 150 days; (c) design of four
financial instruments during the project versus target of three;
(d) Funding of 38 active startup that launched 19 new products
versus 10 and 8, respectively; (e) mentoring of 51 startups versus
initial target of 10; (f) rebuilding staff capacity in spite of two
waves of staff departures; and (g) being recognized by regional
peers.
50. The independent quantitative evaluation (see annex 3 for
analysis) delivered by the external consulting firm in October
2015, titled ‘Estimation of the Effect of Participating in the IF
Financing Programs on Firm Performance,’ concluded that (a) the IF
was a credible and transparent platform, and accessible to many
Serbian businesses, not just a small group of entrepreneurs; (b)
Mini and Matching Grant Programs, developed and piloted within the
project, were found to be clear, accessible, and effective, with
the application process being similar to other international
programs and the evaluation and decision process being transparent
and trustworthy; (c) the IF succeeded in expanding the
opportunities for tech-based entrepreneurs by creating a number of
innovative start-ups/spin-offs. Almost half (156 out of 326) of the
firms that submitted the application for IF financing were created
for them to apply for the IF operation. In addition, the evaluation
found a high survival rate among supported firms. The IF succeeded
in expanding the network of tech-based entrepreneurs by creating a
relatively high amount of collaborations with academia and industry
(also internationally); and (d) under the Mini Grants Program (€2.9
million), 40 firms were approved, of which 37 were active at the
end of the project period (January 2016). Simple differences in
sales changes between firms that received a Mini Grant and firms
that applied but were rejected revealed that the former increased
their sales by an additional €27,000 and their employment by an
additional five employees, on average. Based on these numbers, on
average, the additional sales generated as a result of the Mini
Grants was approximately €1 million. Based on which, the investment
in this component would be recouped in a period of three
years.
51. The RDI TA (€0.7 million), conducted four instead of two
in-depth RDI assessments planned, identified 10 RDIs for potential
institutional improvements support instead of five, delivered on
three technologies transferred, and brought a new commercialization
culture to these RDIs. Most importantly, perhaps, learnings under
this component informed the design of the SRITTP and
C&JP.
3.4 Justification of Overall Outcome Rating
Rating: Satisfactory
Relevance of
Objective
Design
Implementation
Objective
High
Substantial
Substantial
High
Substantial
Satisfactory
52. The overall outcome rating is based on ratings for Relevance,
Efficacy, and Efficiency as broken down in Table 4. Project
Objective remains relevant and aligned with Serbia’s overall
development priorities and is rated as High. Project Design and
Implementation are both deemed to be relevant in the EU
pre-accession context and are therefore rated as Substantial. Some
delays were experienced during early stages of implementation and
there was also a need to extend the closing date of the project.
The delays were justified (and agreed to by the World Bank, donor,
and recipient) given strategic guidance provided to the IF by its
advisory team to stagger the launch of the Mini and Matching Grant
Programs early during implementation to allow for learning to be
incorporated in subsequent rounds. The extension facilitated the
completion of all awardees’ projects, achievement of technology
transfer targets with RDIs, and, most importantly, rebuilt the
ministry’s commitment on innovation policy and RDI reforms. In this
context, given that the Project mostly exceeded the PDO outcome
targets, its Efficacy is rated as High and Efficiency was rated as
Substantial. All of the aforementioned ratings lead to the Overall
Outcome Rating of Satisfactory.
3.5 Overarching Themes, Other Outcomes and Impacts
(a) Poverty Impacts, Gender Aspects, and Social Development
53. This was a small pilot project aimed at stimulating innovative
activities in the enterprise sector. The 51 projects financed
through the pilot in the total amount of €6 million over four years
were too small to have an impact on poverty, gender aspects, or
social development. However, the increased income and employment
generated by the pilot program have given an indication that, if
scaled up substantially, the program could have a positive impact
on poverty reduction and economic development (see annex 3).
54. Women entrepreneurs. The IF programs supported a few remarkable
women researcher become entrepreneurs who were even featured in the
international media. They have become an inspiration for younger
women entrepreneurs. It would be important to strengthen gender
emphasis in future programming given the very specific challenges
Serbian women entrepreneurs face in entering the market, accessing
both debt and equity finance, and extensive harassment while
running a business.
(b) Institutional Change/Strengthening
55. The project has had a significant impact with respect to
institutional changes, especially with respect to the IF, but also
the management teams and researchers at RDIs supported under the TA
program, as well as MOESTD, as discussed above.
(c) Other Unintended Outcomes and Impacts (positive or
negative)
56. At entry, there were few players of any significance in
Serbia’s innovation and entrepreneurship ecosystem, with the
exception of incubators in Belgrade and Novi Sad. By project
closing multiple public and private actors were active in Serbia,
including business angels, accelerators, incubators, technology
transfer offices, science parks, and so on. As the project did not
actively finance ecosystem development, causality cannot be
inferred between the project and its evolution. However,
stakeholders and beneficiaries have credited the IF and its
programming for shining a spotlight on Serbia’s early-stage
entrepreneurial pipeline, attracting regional investors,
highlighting its policy efforts to promote research and enterprise
innovation.
3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder
Workshops
57. Based on the interviews carried out with the beneficiaries as
part of the M&E process, the M&E consultants and World Bank
staff concluded that (a) overall project objectives were adequately
specified; (b) the Mini and Matching Grants Programs were clear,
accessible, and effective; (c) program evaluation and decision
process was transparent and trustworthy; (d) application process
was similar to other programs (internationally); (e) the IF staff
ranked high on professionalism and willingness to help, and were
flexible; (f) implementation was effective—most of the projects
achieved their objectives; (g) RDI management valued the changes in
researcher behavior toward technology transfer; and (h) the
Government takes ownership of R&D sector reform agenda.
4. Assessment of Risk to Development Outcome
Rating: Negligible
58. The risk to development outcome is considered Negligible. All
activities are either completed or sustainable under follow-on
projects. Although there was a change of Government during project
implementation, the risk that the program would be derailed did not
materialize. Sustainability of the pilot’s achievements has been
further enhanced by the Government and the World Bank’s commitment
to further support the PDO through the SRITTP and C&JP.
5. Assessment of Bank and Recipient Performance
5.1 Bank Performance
Rating: Satisfactory
59. The World Bank’s performance for quality at entry was rated
Satisfactory. The World Bank heavily leveraged top-notch global
expertise and lessons learned from multiple post-transition
economies to recast the innovation policy discourse to focus on the
role of the Government and entrepreneurship in building that
national innovation system. Additionally, it emphasized the
importance of piloting and stronger than usual M&E practices
for this institutional capacity-building project.
(b) Quality of Supervision
Rating: Highly Satisfactory
60. Quality of supervision was far more intense than a regular
World Bank project and very meticulous given the pilot nature of
the project and first time relationship with the donor. Supervision
missions were carried out at least twice a year and up to five
times in the early stages of the project owing to the flexibility
afforded by the World Bank-executed TA component. The World Bank
team ensured continuity in project management and a strong thematic
and operational skills mix with strong support from staff in the
local office. The World Bank-executed component enabled
mobilization of leading global practitioners in their respective
fields, thus earning the trust and respect of the research
community. The team was proactive and responsive to the multiple
GoS requests for change during project implementation and
demonstrated flexibility and understanding in the use of project
resources. The World Bank led project restructuring in consultation
with the donor and recipient to maximize achievement of the PDO. On
fiduciary requirements, the performance was Highly Satisfactory
with FM, procurement, and environment activities under the project
implemented effectively with significant capacity building and no
major issues.
(c) Justification of Rating for Overall Bank Performance
Rating: Satisfactory
61. Overall, the World Bank was very proactive and responsive to
the client’s needs from the design stage through to project
closing, with close supervision and excellent support from the
country office. The World Bank team also played a very important
role in establishing the IF’s capacity by offering its own
experience and guidance. Based on the above, the overall World Bank
performance in ensuring quality at entry and quality of supervision
is rated Satisfactory.
5.2 Recipient Performance
(a) Government Performance
Rating: Moderately Satisfactory
62. The recipient worked closely with the World Bank during project
preparation in a satisfactory manner. During implementation, the
recipient could have been more proactive in providing support to
the IF (especially its operating budget), and introducing some
quick win policy recommendations resulting from the RDI TA program.
Although the Recipient complied with all covenants under the Grant
Agreement and development outcomes were met by the project, the
shortcomings in budget support to the IF slowed down implementation
progress in early 2014 and destabilized the IF’s operations.
Therefore, the overall performance of the Recipient is rated as
Moderately Satisfactory.
(b) Implementing Agency or Agencies Performance
Rating: Highly Satisfactory
63. The IF as the implementing arm of the Recipient was very
effective in providing efficient implementation, monitoring, and
reporting, allowing for better management and implementation of the
project and facilitating the World Bank’s supervision activities.
The IF was staffed with qualified experts, selected through a
competitive process and adequately compensated for their work.
Although there were teething problems such as the absence of a
permanent managing director and lack of timely GoS budget support
on multiple occasions, the IF staff have done a remarkable job in
implementing the project and achieving the PDOs. The IF staff
should be commended for the professionalism exhibited in the
discharge of their duties and the performance is rated Highly
Satisfactory.
(c) Justification of Rating for Overall Recipient Performance
Rating: Satisfactory
64. The overall Recipient performance is rated Satisfactory. The
success of the project was to a large extent due to the efficiency
and professionalism demonstrated by IF staff in the implementation
of the program and they should be highly commended. The Recipient,
although committed, could have been more proactive in providing
budget and policy support to the IF, RDIs, and the sector
overall.
6. Lessons Learned
65. Governance arrangements are key for institutional integrity and
program delivery. Governance mechanisms established at the IF were
underscored by the transparent selection process that placed a
premium on merit and commercialization, and integrated feedback
from international peer reviewers, decision making by an
independent IC and continuous guidance from a professional global
advisory team. This positioned the IF as an institution with
significant credibility and capacity to absorb and distribute EU
funds in a thorough and independent manner. This aspect was
confirmed by beneficiaries, who commended the transparent and
independent process and the mentoring received from peer
reviewers.
66. Institutional capacity matters for delivery of innovation
programming. A well-functioning national innovation agency, staffed
with competent and committed people and fairly remunerated is
essential for implementation of a complex project, such as this,
and is an important element of the national innovation system. The
IF is an institution that is well regarded by its beneficiaries and
its peers, and should be nurtured to continue to serve the country
and remain agile in an ever changing global innovation landscape.
This will mean additional rounds of capacity building and staff
renewal to meet new client and market demands. This should remain
part of the IF’s strategic planning efforts.
67. Horizontality of innovation policy and programming matters in
nascent ecosystems. One of the principles of design considered
important in nascent post-transition innovation
ecosystems—dominated by entrenched state enterprise and RDIs—is
horizontality of the innovation policy and interventions, that is,
sector agnosticism. At entry, many assumed that the deal flow would
be limited to innovative projects in ICT and digital technology
from Novi Sad and Belgrade. The pilot—by adhering to horizontal
selection criteria that emphasized merit and commercialization
potential—revealed a diverse pipeline in terms of sectors, cities,
and gender.
68. Size of innovation effort matters and so do rhythm and scale.
The 51 projects the IF financed over four years was too small a
number to have an economy-wide impact, but manageable for capacity
building. However, to move the needle at an economy-wide level, the
independent evaluation estimates that the IF would need to provide
funding to 200–300 projects annually. Having established the IF and
piloted instruments that demonstrate additionality for enterprise
innovation, the GoS should ensure continuous uninterrupted support
to retain IF capacity, generate innovative entrepreneurs, rebalance
the enterprise innovation system, and continue to position Serbia
as a budding knowledge and IP creator in the global innovation
landscape. Losing the momentum generated would be ill advised and
create significant loss of trust in public institutions and could
undermine future efforts.
69. Visibility of innovation activities really matters. World Bank
operations could take a serious lesson from the EU’s approach and
benefit from a more systematic approach to providing visibility to
project activities with stakeholders, sharing lessons learned, and
celebrating successes. The Project Opening Ceremony engaged the
IF’s advisory and management teams, and its IC to showcase the
capacity engaged and discuss intentions transparently, thus
building credibility among the sceptical research and
entrepreneurial communities. The Closing Ceremony—a celebration of
the journey the Serbian ecosystem had made over the course of the
project—displayed the institutional capacity of the IF and RDIs,
the remarkable 50 plus entrepreneurs who demonstrated their firms’
prototypes and viable products, as well as the investor groups and
entrepreneurship support institutions that did not exist at the
onset of the project. The unrehearsed testimonials provided by
these ecosystem actors during visibility events were picked up by
policy makers and donors and are informing future decisions on
financing.
70. Support for the evolution of an innovation ecosystem is
critical. In the early days of the project, the IF was a lonely
advocate for the ecosystem and would often come under undue
scrutiny and criticism. Eventually the strong entrepreneurial
pipeline attracted regional partners and global investors, making
it easier to advocate for support to other actors and promote
long-term symbiotic relationships in the ecosystem.
71. The policy world is changing fast. The innovation arena is
witnessing an increase in well-networked private sector investors
and support institutions globally, including in incubation,
acceleration, and financing. Policy makers are beginning to review
policies and adopt legislation to enable crowd funding, which could
further democratize early-stage innovation finance. While not an
immediate concern for any of the post-pilot operations, the GoS and
IF should familiarize themselves with these developments and policy
implications to ensure relevance of future design and
implementation.
72. Flexibility during design and implementation are key for
pilots. During design and especially during implementation of this
pilot, the World Bank team worked closely with the IF to monitor
project progress. When issues were encountered, the teams worked
quickly to find solutions. This flexibility allowed for smoother
and efficient implementation. A few examples include (a) design of
the mentoring program to accommodate mandatory and customized
training elements for IF staff and beneficiary entrepreneurs to be
more useful; (b) deploying the Mini and Matching Grant programs
sequentially so the IF could incorporate lessons from the first
call of the Mini Grant Program for subsequent programs; (c)
increasing the private sector contribution (from 25 percent to 30
percent) under the Matching Grant program; (d) converging on design
details for the royalty scheme; and (e) engaging commercialization
brokers to work with RDIs to identify commercialization wins within
the lifetime of the project.
73. Incentives Matters: Entrepreneurs must have “Skin in the Game”.
The requirement for the beneficiary entrepreneurs to provide a
match in case only (not in kind) and the success royalty component
weeded out several serial ‘grant writers’ during the project. This
scenario is known from international practice and hence influenced
project design and played out in Serbia.
74. Building relationships with development partners takes times
but is worthwhile. This was the first time in Serbia that a Trust
Fund Agreement was forged between the Bank and the EU Delegation.
As both organizations were unfamiliar with policies and procedures
of their counterparts, this led to some delays during project
preparation. However, once both organizations were comfortable with
each other’s policies and procedures, project implementation
progressed smoothly. It would be prudent that for future joint
collaborations to allow some lead time for project preparation so
that any procedural issues could be sorted out and not delay
project preparation. Additionally, the Bank team ensured that it
met with the EUD counterparts during each mission to allow maximum
information sharing on project progress and any challenges that
could be jointly addressed in bilateral conversations with the
GoS.
7. Comments on Issues Raised by Recipient/Implementing
Agencies/Partners
(a) Recipient/implementing agencies: please see Annex 7 for report
and statement from IF.
(b) Co-financiers: please see Annex 8 for statement from EUD.
(c) Other partners and stakeholders: Please see Annex 8 from key
partner: Belgrade Incubator
Annex 1. Project Costs and Financing
(a) Project Costs by Component (in Euro)
Appraisal Estimate
3.21
1.11
2.10
35
6.00
6.00
0.0
100
Component 3. Provision of Technical Assistance to Research and
Development Institutes (RDIs)
0.70
0.70
0.0
100
0.59
3.82
1.06
2.76
28
6.51
5.41
1.1**
83
Component 3. Provision of Technical Assistance to Research and
Development Institutes (RDIs)
0.80
0.80
0
100
0.59
7.86***
Note: *Government financing covering period from 2011–2015 (funds
are expressed in € using middle exchange rate of National Bank of
Serbia on the last day of each year). Initially, project closing
date was set on November 30, 2014. However, it was extended to
January 10, 2016, which caused increased Government financing
(Government financing in 2015 equals around €534,000).
**Contributions from the Mini and Matching Grant private sector
enterprises were at least €1.1 million.
*** EURO 0.655 million was cancelled.
(b) Financing
Annex 2. Outputs by Component
Component 1: Capacity Building of the Serbia Innovation Fund -
Recipient Executed
1. The IF was to be established with the objective of encouraging
entrepreneurship and managing financing for innovation, as well as
participating in co-financing of programs, projects, and other
activities organized by international organizations, financial
institutions, and the private sector. This has been fully achieved.
Under this component, the IF developed the requisite capacity for
designing and managing grant instruments geared toward
technology-based firms. Importantly, the IF has built and retained
an impressive international team, including strategic, operations,
and regional advisors, in addition to strong governance practices
espoused by its international diaspora-based independent IC.
2. The major networking and educational events held by the IF were
project launch and launch of the Mini Grants Program, followed by
the launch of the Matching Grants Program, the IP workshop (for
RDIs and key stakeholders), WB EDIF workshop (for RDIs and private
sector and key government stakeholders), and four info-days for the
awardees (addressing project implementation and training held by
Serbian IP office), presentation of the IF Annual Report, knowledge
economy workshop, industry and RDI consultation regarding the new
technology transfer project design, consultations with local
Technology Transfer Offices on the cooperation and functioning of
centralized TTF.
3. The IF also provided several training programs for its staff and
beneficiary enterprises. One of the essential services the IF
provided to its grant awardees was Enterprise Training, which was
conducted by international advisors covering topics of innovation,
business formation and expansion, IP protection and
commercialization, investment readiness, and other valuable themes
associated with early-stage technology and business development. As
part of its program, the IF also conducted several project
visibility events and workshops. Overall, this component is
considered highly satisfactory for the establishment of the IF,
with its remarkable governance and the institutional
capacity.
4. Staff training for the IF. The following training courses were
attended by IF staff.
· Preparation and management of the World Bank-funded projects
(World Bank office, Belgrade). IF staff received training on all
aspects on the preparation and implementation of World Bank-funded
projects.
· Management of innovation and technology support programs (at
BICRO, Croatia). Training provided at BICRO was expected to
introduce IF staff to BICRO programs and organization, management
of calls for financing, due diligence, M&E. Introduction to
programs for supporting operation of incubators and technology
transfer programs was also provided.
· FM (Turin, Italy). Training was provided to the financial manager
to design and operate the FM systems and execute corresponding
payments/disbursements in accordance with sound professional
standards and the harmonized requirements of donors and their
respective governments.
· Venture capital (VC, Tel Aviv, Israel). Training was provided to
acquire critical skills in five key areas: (a) interpreting
governance diagnostics and political economy analyses, (b) creating
multi-stakeholder coalition building strategies and tactics to
support reform, (c) providing communication skills that support the
implementation on governance reforms, (d) leveraging social/digital
media tools and analytics effectively, and (e) developing
communication metrics and applying M&E frameworks. Training was
attended by the IF’s interim managing director.
· VC (Tel Aviv, Israel - two times for staff). Training consisted
of introduction to organization, operations, and principles of
private sector VC funds, transaction, and due diligence process for
early-stage financing. Training was useful for the IF managing
director and program managers to structure and implement privately
managed/publically funded equity instruments and to efficiently
connect Serbian companies with private sector VC funds.
· World Bank procurement and FM procedures (Sarajevo, Bosnia and
Herzegovina). Introduction to the World Bank procurement and FM
procedures and guidelines was provided. Training was attended by
the procurement associate and financial manager.
· Innovation funding programs (Helsinki, Finland). Tailor-made
training for the IF managing director and program managers at TEKES
and SITRA in Finland was provided to get an overview of financing
mechanisms for private sector companies and enterprise R&D as
well as fund management in Finland; applications evaluation;
monitoring, funds management; and programs for large
companies.
· Program management (Turin, Italy). Two-week training on project
and program cycle management based on the logical framework
approach, including project identification, stakeholder analysis,
project design, and the development of M&E systems was
provided. Training was attended by the IF’s senior associate.
· Represent. The IF’s public relations (PR) person and a person in
charge of managing the social network pages (LinkedIn and Facebook)
have received training to provide knowledge on the most important
aspects of presenting the institution through social network
profiles/pages.
· Training on EU PRAG procedures. IF employees have attended the
tailor-made training designed to provide better understanding of
the PRAG procedures that was required in the implementation of the
projects financed by IPA funds.
· PR training. IF employees received PR training from the World
Bank representative (Belgrade office) in Belgrade.
5. A critical ingredient for the success of the project was that a
responsible governance mechanism was established at the IF,
underscored by the transparent selection process that placed a
premium on merit and commercialization potential of projects, and
included inputs from international peer reviewers and decision
making by an independent IC. This has positioned the IF as an
institution with significant credibility and capacity to absorb and
distribute EU funds in a professional and independent manner.
Component 2: Implementation of Financial Instruments Supporting
Enterprise Innovation - Recipient Executed
6. The IF started its operation on December 5, 2011, as Serbia’s
main agency for innovation support through two main programs: the
Mini Grants Program and the Matching Grants Program.
7. The purpose of the Mini Grants Program was to expand
opportunities for tech-based entrepreneurs by stimulating
commercialization of R&D and the creation of innovative
enterprises based on knowledge through start-ups and/or spin-offs.
The Matching Grants Program aimed to expand collaboration
opportunities for innovative micro and small companies with
strategic partners (for example, private sector industry, R&D
organizations, and VC/private equity funds) with the goal of
increasing private sector investment in R&D and the
commercialization of projects promoting new and improved products
and services.
8. The financing offered by the Mini Grants Program covered a
maximum of 85 percent of the total approved project budget and up
to €80,000 for a one-year project. The reminder of the total
approved project budget was to be secured by the applicant from
other, preferably private sector sources, independent of the
IF.
9. The financing offered by the Matching Grants Program covered a
maximum of 70 percent of the total approved project budget and up
to €300,000 for a two-year project. The remainder of the total
approved project Budget was to be secured by the applicant from
other sources independent of the IF such as private sector
industry, private investors/VC/private equity funds or the
applicant’s own internal resources.[footnoteRef:7] [7: Upon
successful technology commercialization, the revenue (and not just
the profits) derived from the sales of the product/service and any
or subsequent products/service based on the technology developed in
the Matching project financed by the IF, become the basis for a
royalty payment in compensation for the IF financing, at a rate of
5 percent of annual revenue derived from sales of product or
service or 15 percent of licensing revenue derived from such
product/service up to 120 percent of financing received from the IF
for a period of up to five years after project completion,
whichever is achieved first.]
10. Financing decisions were made on a competitive basis by the
IF’s independent IC. The number of awards was determined by the
quality of the proposals and subject to the total funds available
(allocated) to the programs. In addition, these programs provided
support for FM, IP protection, and business development training,
as well as mentoring of the awardees to enable them to prepare for
the next stage of development.
11. Between January 31, 2012 and October 28, 2013, the IF managed
four financing calls during which 326 Serbian enterprises submitted
applications for either the Mini or Matching Grants Programs. The
IF held about 20 visibility events per call to promote the CFPs and
to explain the Mini and Matching Grants Programs. Some 91 firms
applied to the Matching Grants Program, while 247 firms applied to
the Mini Grants Program (12 firms applied to both the Matching and
Mini Grant Programs). Overall, 471 projects were submitted by the
326 firms. A total of 55 projects were approved for IF support, out
of which 52 projects were financed (out of the 471 projects); a 12
percent approval rate. Details of the selected