Sdf

Ymchwil gymdeithasol

Social research

Rhif/Number:

Mid-term Evaluation of the

SMARTCymru RD&I Financial

Support for Business Programme

Mid-term Evaluation of the SMARTCymru RD&I Finance for Business Programme

CM International

N.B. English text in curly brackets for English version only Welsh text in curly brackets for Welsh version only (swap positions with English text)

(Views expressed in this report are those of the researcher and not

necessarily those of the Welsh Government) (Eiddo’r ymchwilydd yw’r farn a

fynegir yn yr adroddiad hwn ac nid o reidrwydd farn Llywodraeth Cymru).

For further information please contact / I gael gwybodaeth bellach cysyllter â;

Joanne Coates

Social Research and Information Division

Welsh Government / Llywodraeth Cymru

Cathays Park

Cardiff

CF10 3NQ

Tel/Ffôn: 02920 825540

Email/Ebost: Joanne.Coates@wales.gsi.gov.uk

Welsh Government Social Research, 2014 / Ymchwil Gymdeithasol

Llywodraeth Cymru

ISBN

© Crown Copyright / Hawlfraint y Goron

Table of contents

Glossary of acronyms 2

Acknowledgements 3

List of tables 4

List of figures 5

Summary 6

1 Introduction 11

2 Programme logic model 15

3 The programme context 22

4 The programme profile 30

5 Outputs and impacts 41

6 Value for money 59

7 Conclusions and recommendations 67

Annex 78

1

Glossary of acronyms

A4B Academic Expertise for Business

BETS Business, Enterprise, Technology and Science - the former Welsh

Government Department, currently known as Economy, Science and

Transport

BIP Business Innovation Programme

ED Experimental Development

ERDF European Regional Development Fund

ERP Economic Renewal Plan

EST The Welsh Government department for Economy, Science and

Transport

EU European Union

FP7 Seventh Framework Programme

FTE Full Time Equivalent

GDP Gross Domestic Product

GVA Gross Value Added

HEI Higher Education Institution

ICT Information and Communication Technologies

IP Intellectual Property

IR Industrial Research

R&D Research and Development

RD&I Research, Development and Innovation

SME Small and medium sized enterprise

TCF Technical and Commercial Feasibility

TSB Technology Strategy Board

WEFO Welsh European Programme Office

WG Welsh Government

Acknowledgements

This report has been written by Dr Dylan Henderson (project manager and

lead researcher), Meirion Thomas (project director) and Ioan Teifi

(researcher) at CM International.

Important contributions have also been made by the Welsh Government’s

Innovation team.

List of tables

Table 1. SMARTCymru intervention rates ...................................................18

Table 2. SMARTCymru target indicators .....................................................19

Table 3. SMARTCymru programme funding ...............................................20

Table 4. Workplace based GVA at current basic prices ..............................23

Table 5. Business expenditure on R&D.......................................................24

Table 6. Mid-term programme cost – Convergence ....................................37

Table 7. Mid-term programme cost – Competitiveness ...............................39

Table 8. Convergence output indicators ......................................................42

Table 9. Competitiveness output indicators.................................................42

Table 10. How important is the outcome of the project for the company?..44

Table 11. To what extent were the project objectives met?........................45

Table 12. Prospect of commercialisation....................................................45

Table 13. Protection of IP...........................................................................46

Table 14. Satisfaction with Welsh Government's support in developing

projects.......................................................................................47

Table 15. Action taken in projects had SMARTCymru funding not been

available .....................................................................................48

Table 16. Other sources of funding sought had SMARTCymru not been

available .....................................................................................49

Table 17. New products/process/services as a result of SMARTCymru ....50

Table 18. Have you invested more in R&D as a result of the project? .......51

Table 19. Extent to which SMARTCymru encouraged/improved capability to

apply for competitive R&D funds in companies ..........................51

Table 20. Maintenance of linkages and collaborative relationships post-

SMARTCymru ............................................................................52

Table 21. What impact has the SMARTCymru programme had on your

company's attitude to collaboration ............................................53

Table 22. Comparative deadweight assessments......................................55

List of figures

Figure 1. Mid-term evaluation work stages................................................13

Figure 2. SMARTCymru programme logic model......................................21

Figure 3. SMARTCymru programme applications .....................................25

Figure 4. Geographic location of projects – Convergence area.................30

Figure 5. Geographic location of projects – Competitiveness area ...........31

Figure 6. Participant company turnover – Convergence area ...................32

Figure 7. Participant company turnover – Competitiveness area ..............33

Figure 8. Employees – Convergence area ................................................33

Figure 9. Employees – Competitiveness area ...........................................34

Figure 10. Sectors – Convergence area......................................................34

Figure 11. Sectors – Competitiveness area.................................................35

Figure 12. RD&I Phase – Convergence area ..............................................36

Figure 13. RD&I Phase – Competitiveness area .........................................36

Summary

1. This report sets out findings from the Mid-term evaluation of the

SMARTCymru Research Development and Innovation (RD&I) financial

support for business programme, and provides an analysis of

achievements to date, and recommendations for its remaining period of

finding.

2. The evaluation research was undertaken in the period June to October

2013 by CM International, and is based on a management and delivery

workshop and interviews, analysis of monitoring data, and interviews with

a sample of businesses.

Programme model

3. The SMARTCymru programme offers all-Wales support for businesses at

different stages of the RD&I process, including Technical and Commercial

Feasibility (TCF), Industrial Research (IR), Experimental Development

(ED), and Exploitation. It represents the Welsh Government’s core support

for business RD&I, and operates alongside other Welsh Government

programmes such as the Business Innovation programme and the

Academic Expertise for Business (A4B) programme.

4. The results of this evaluation suggest that the programme has been

established with a sound programme logic model, based on an identified

need, clear objectives, grant activities focused on the different RD&I

phases, and anticipated targets both informed by early programme

experience, and aligned to the objectives. Clear links are also embedded

in the model, to maximise the potential for follow-on support, access to

academic expertise and so on.

5. The programme is intended to contribute towards the cross-cutting

themes, through the referral to sources of expertise in the all Wales

Regional Centre Services (now Business Wales Centres).

Programme context

6. The findings from the mid-term evaluation suggest the SMARTCymru

programme has faced significant challenges resulting from the economic,

policy and organisational context. The economic downturn, and associated

decline in business R&D expenditure contributed to lower than expected

demand for the programme. This was compounded by the changes

brought about by the implementation of the Economic Renewal

Programme (ERP), notably the decision to move towards repayable

finance in place of grants and the reorganisation of the then BETS

department into sector teams. Together these factors hindered the

Innovation team from effectively promoting the programme, and resulted in

confusion, both internally and externally, about the availability and offer of

the programme.

7. While the confusion surrounding the programme’s availability and offer has

subsided with the decision to revert back to grant funding, (re)adopting the

SMART branding, and relocating the Innovation Specialists (back) to

within the Innovation team, the challenge for the programme continues to

be one of stimulating demand in the remaining part of the programme. The

enhancement of the links to potential referral programmes and

organisations indicate one of the ways in which this challenge has been

addressed.

Programme profile

8. At the mid-term stage SMARTCymru has supported 116 projects (84 in the

Convergence area, and 32 in the Competitiveness area). These projects

are typically with SMEs, with a number of key priority sectors represented,

including sectors such as the Advanced Manufacturing and Materials

sector, the ICT Sector, Life Sciences, and Financial Services. All are

consistent with the SMARTCymru quality criteria – defined by a robust

technical and financial due diligence processes.

9. With regard to one key element of the cross cutting themes, anecdotal

evidence provided by the Innovation team suggests that the equality and

diversity profile of participants is skewed towards ‘white, middle aged,

male’ participants.

10.Programme costs are below the forecasts at the mid-term. This can largely

be explained by the difficulties faced in adapting to the policy and

organisational changes associated with the ERP process, on one hand,

and the economic downturn/recovery on the other.

11.Programme costs are also below what was expected at the start of the

project, however, the results of the evaluation highlight the important

challenge of encouraging more companies to progress to the later RD&I

phases.

Programme outputs and impacts

12.The outputs and results collected for the SMARTCyrmu programme at the

mid-term state suggests that the programme is some way behind the

original forecasted profile targets. Results for Convergence area exceed

those of the Competitiveness area, with 85 per cent of the profiled targets

for gross jobs created and 61 per cent of the investment induced profile

target (to date) achieved. The number of (Convergence) enterprises

financially supported is, however, some way below the profiled targets,

with only 27 per cent of the delivery profile target achieved to date. It

should be noted, however, that the profiled targets reflect assumptions

made at the start of the process, and do not take account of the decline in

applications brought about by the economic downturn and the ERP

process.

13.Programme impacts at the mid-term stage are both economic- and

innovation-related. Importantly, based on findings from the business

interviews (10), these impacts have been achieved with a low level of

deadweight (30 per cent). They also suggest that Welsh Government is

funding projects that are strategically important to the companies.

14. Innovation impacts, at the mid-term, are limited by the early stage nature

of the RD&I projects. The business interviews do, however, point to the

majority of companies both meeting the technical objectives, and

anticipating good commercial potential for the new/improved, products,

processes and services. Companies are also investing more in R&D,

improving their capability to secure additional funding from public support

programmes. While the potential for innovation spillovers is also evident,

these are likely to take time to fully develop.

15.The economic impact results suggest that the programme results to date

will contribute 29 net additional jobs and a cumulative net total of £4.5m

GVA based on the estimated persistence of benefits achieved at the mid-

term (£1.7 actual)1. It should be noted, however, that the main aim of the

See Annex for more details on these net additional economic impact calculations. 1

programme has not been to create jobs but to increase R&D activity, and

develop new products, processes and services.

16.This performance equates to estimated return on investment, based on

cumulative net additional impacts, of:

£1 SMARTCymru total programme spending produces £0.44

GVA to the Welsh economy

£1 SMARTCymru Welsh Government spending produces £5.62

GVA to the Welsh Economy

SMARTCymru programme cost per project of £87,800

17.The Return on investment (based on total programme spending) is

consistent with comparator programmes such as the Invest NI Start

scheme, although the results of an earlier evaluation of SMARTCymru

suggest that this impact is likely to be greater at the final evaluation point.

Recommendations

18.There are a number of areas where the management and delivery of the

programme could be strengthened in the remaining phase of the project:

a. Recommendation 1: Welsh Government should continue to review

longer term innovation impacts through its Innovation Impacts

programme, complementing the regular SMARTCymru programme

evaluation.

b. Recommendation 2: Welsh Government should prioritise demand

stimulation measures, including supporting companies to move on

to the later RD&I phases in a timely manner.

c. Recommendation 3: Welsh Government should seek to ensure

businesses and stakeholders are aware that SMARTCymru is

available and open to applications.

d. Recommendation 4: Welsh Government should consider the

options available for SMARTCymru to revise the final and profiled

performance targets, with a focus on prioritising projects in the ED

and Exploitation phases.

e. Recommendation 5: Welsh Government should seek to ensure

delivery efficiency is maximised through appropriate integration

across the SMARTCymru management and delivery processes.

f. Recommendation 6: Welsh Government should review its support

for innovation projects where ICT plays an important enabling role.

g. Recommendation 7: Welsh Government should consider

refocusing funding for the Exploitation phase.

h. Recommendation 8: Welsh Government should explore the

potential to strengthen synergies and knowledge exchange with key

stakeholders such as Finance Wales.

i. Recommendation 9: Welsh Government should establish robust

procedures to support companies in the area of the cross-cutting

themes and the ongoing monitoring of results.

1 Introduction

1.1 The SMARTCymru Research, Development and Innovation (RD&I)

Financial Support for Business Programme (the programme) was

launched2 in the Convergence region of Wales in 2010, and

subsequently rolled out in the Competitiveness region in 2011. It offers

all-Wales support for businesses at different stages of the RD&I process,

including Technical and Commercial Feasibility (TCF), Industrial

Research (IR), Experimental Development (ED), and Exploitation. It

represents the Welsh Government’s core support for RD&I, and

operates alongside other Welsh Government programmes such as the

Business Innovation programme and the Academic Expertise for

Business (A4B) programme.

1.2 The programme was initially managed as part of the Single Investment

Fund, but is now managed by the Economy, Science and Technology

(EST) Innovation team, with the support of Welsh Government

Innovation Specialists providing all Wales coverage.

1.3 This report sets out evaluation findings for both the Convergence and

Competitiveness programmes at the mid-term stage (June, 2013).

Aims of the evaluation

1.4 The focus of this evaluation is to understand the mid-term performance

of the SMARTCymru programme against both its objectives and

specified targets. It is intended that the results of the evaluation will feed

into the future direction of the programme in the remaining phase.

1.5 The objectives of the evaluation, taken from the Specification, are to:

To provide a mid-term evaluation report for the SMARTCymru, RD&I

programme that assesses the programme in relation to its progress

towards the criteria outlined by the Welsh European Funding Office

(WEFO) by considering:

o The progress towards the project’s long-term aim, objectives

and targets.

2 It was originally known as the RD&I Financial Support for Business Programme.

o How the project is contributing to the cross-cutting themes

(equal opportunities and environmental sustainability).

o The need and demand for the grants provided through the

project.

o The quality of businesses supported and created by the project,

including growth potential, ability to create jobs and

sustainability.

o The integration with other provision for businesses, including the

New Business Start-Up Support and JEREMIE projects;

o Any evidence of ‘behavioural additionality’ as defined in the

business plan.

o The appropriateness of the project indicators and related

targets, including an assessment of potential achievements after

the Project End Date.

o The impact of grants provided through the project.

o The extent to which the project is contributing to structural

change and sustained impact beyond the funding period

including the move from a grant to an investment culture.

o Progress towards an exit strategy.

Source: adapted from Welsh Government, n.d.

1.6 The evaluation objectives also include a number of ‘considerations’ set

out within the programme’s Grant Award Letter from WEFO.

What has been done by the programme and how has this been

understood by those involved?

How has the programme been delivered and what outcomes have

been achieved? This also includes an assessment of the monitoring

data collection processes and monitoring processes for softer

outcomes.

What barriers and constraints has the programme faced? This includes

consideration of the internal and external influences on the programme

during its initial stages (e.g. policy, organisational and staffing changes

within the Department, recessional effects) and analyse their effect on

achieving targets and outputs.

To consider what are participants gaining from the programme and

how is this different from what they would have done anyway? This

includes consideration as to how businesses are exploiting and

commercialising the technologies financially supported through this

programme.

To identify examples of best practice and lessons learned, with

evidence based recommendations for taking the programme forward.

Source: adapted from Welsh Government, n.d.

Research methodology

The mid-term evaluation is based around five work stages and builds largely

on monitoring data collected by the Innovation team at the Welsh

Government, alongside a small number of interviews with businesses and a

workshop with management and delivery staff. A summary of the approach

adopted can be found in figure 1 below detailing the focus and scale of the

fieldwork undertaken for the evaluation.

Further details of the approach adopted can be found in Annex I, including

details of the approach and impact methodology adopted.

Figure 1. Mid-term evaluation work stages

1.7 The mid-term evaluation research was undertaken in the period June to

October 2013.

Structure of the report

1.8 The findings in this report are set out as follows: section 2 provides an

overview of the SMARTCyrmu programme logic model and its

underpinning assumptions (2), followed by a profile of activity at the mid-

term based on activity and characteristics of the beneficiaries (3). The

report then considers the policy and operational context of the

programme (4). The main focus of the report, however, is found in

sections 5 and 6, where mid-term impacts of the SMARTCymru

programme and its value for money are considered. The report

concludes by examining the key mid-term evaluation findings, including

key issues and recommendations for future delivery (7).

2 Programme logic model

2.1 The following sections outline key features of the programme logic

model including its aims, rationale, activities, management and targets

(output, outcome and impact).

Programme aims

2.2 The SMARTCymru aims for both the Convergence and Competitiveness

programmes are shared. They include:

Create high quality R&D related jobs.

Increase business expenditure on R&D through the provision of

financial support to undertake innovative research and technological

development with commercial potential, leading to new products,

processes and technologies.

Encourage and support industry collaborations with other partners with

research-based organisations in carrying out industrial research and

experimental development activities.

Measure the return on investment of supported R&D projects and

provide continuous improvement of the project management of R&D in

companies supported.

Enable business links to other business support to optimise

commercialisation.

Sources: (Welsh European Funding Office, 2012a; Welsh European Funding Office,

2012b)

Programme rationale

2.3 The economic rationale for the SMARTCymru programme is set out in

the related Business Plans prepared for the Convergence (Welsh

European Funding Office, 2012a) and Competitiveness (Welsh

European Funding Office, 2012b) programmes. These documents

highlight the role of business R&D in producing new product, processes

and services, and as a driver of economic development. Wales,

however, has long underperformed in business expenditure on R&D

statistics, accounting (at the time of the Convergence business plan

analysis 2010) for some two per cent of total UK business R&D

expenditure. This figure, it is noted, is some way below what might be

expected given Wales’ population share in the UK (5 %). Providing

financial support for business R&D, it was argued, would further help to

address UK targets for business R&D to reach a total of three per cent of

Gross Domestic Product (GDP).

2.4 The Community Innovation Survey (Eurostat) provides data on the

innovation performance of European member states. This points to key

barriers to innovation in the UK, including the lack of finance, new ideas

and time to innovate, many of which are likely to be shared by Wales.

2.5 The strategic rationale for SMARTCymru is also highlighted in the

SMARTCymru Business Plans (Welsh European Funding Office; 2012a,

Welsh European Funding Office, 2012b). This points to SMARTCymru’s

role in supporting the ‘Innovation and R&D Strategic Framework’ (Welsh

Assembly Government, 2008) objective to ‘increase investment in

commercially driven R&D, focussed on the commercialisation of

knowledge and Intellectual Property’. It will also seek to contribute

towards the strategic objective to ‘support knowledge based/technology

companies, within Wales’, and ‘encourag(e) more to move to Wales’.

The encouragement provided by the programme for collaborative

engagement with higher education researchers further suggests there is

potential to contribute towards the second objective of the Innovation

and R&D framework to ‘maximise the economic impact of academia and

business through technology transfer and the creation of a stronger

science, engineering and technology base with clear commercial

potential’.

2.6 The programme also contributes to Wales’ (then) primary economic

development strategy - the Economic Renewal Plan (ERP) (Welsh

Government, 2010). This document stated that:

‘… Wales must move towards a more R&D intensive and

knowledge-based economy where the right conditions exist for

innovation to flourish.’

2.7 The lack of a large public or private R&D base is cited by the ERP as an

important barrier to improving Wales’ performance in the UK business

R&D ‘league table’. This highlights the importance of encouraging SMEs

to both recognise the importance of R&D, and helping to build their

capability to conduct R&D. These are challenges that the SMARTCymru

programme seeks to address directly.

Programme activities

2.8 The SMARTCymru programme provides match funding to innovative

businesses to ‘develop new, technologically innovative products and

processes with commercial potential’ (Welsh European Funding Office,

2012a). Financial support available according to the stage that the

company has reached in its innovation process. This includes:

TCF (Technical and Commercial Feasibility) study – funding for

SMEs to investigate and assess technological and commercial viability

of new, innovative ideas for products, processes or technologies.

IR (Industrial Research) – funding to support research to acquire new

knowledge, which will facilitate the development of new products,

processes or technologies, or significant improvement of existing ones.

ED (Experimental Development) - funding to implement the results of

industrial research for the development of new products, processes or

technologies, including the creation of results such as a pre-production

prototype, conceptual formulation, product design, and initial

demonstration of pilot projects.

Exploitation – funding to assist with the exploitation costs of a new

product or process (developed in an earlier SMARTCymru funded

phase), including marketing, publicity, advertising, publication of sales

literature, trade fairs and product certification.

Source: (Welsh Government, 2013)

2.9 The published intervention rates for both small, medium and large

enterprises are summarised in the table below:

Table 1. SMARTCymru intervention rates

Small

enterprise

Medium

enterprise

Large

enterprise

Funding

limits

Technical &

Commercial Feasibility

Up to 75% Up to 75% Up to 65% £15,000

Industrial Research Up to 70% Up to 60% Up to 50% £100,000

Experimental

Development

Up to 45% Up to 35% Up to 25% £200,000

Exploitation Up to 50% Up to 50% Up to 50% £20,000

Source: (Welsh Government, 2013)

Programme outputs and impact targets

2.10 The SMARTCymru Business Plans identify the following intended

outputs for SMARTCymru (Welsh European Funding Office, 2012a;

(Welsh European Funding Office, 2012b):

job creation

increase in Gross Value Added (GVA) and turnover

increased business expenditure on R&D

introduction of new products, processes and technologies

industry collaborations with other industrial partners and with

research based organisations in carrying out the industrial research

and experimental development activities

links to other business support to optimise commercialisation of new

products and processes

2.11 Alongside these core output areas, the Business Plans further recognise

the potential for longer-term outcome benefits to emerge, including

encouraging businesses to go on to conduct and attract public and

private sources of funding for innovation, and positive behavioural

changes to R&D (so-called ‘behavioural additionality’), and commercial

value.

2.12 Anticipated impacts from the programme are expressed primarily in

economic terms, and include:

(increase) in company level GVA

change (Increase) in turnover of enterprises supported

number of jobs created

sales of products, processes or service supported by this Fund

jobs safeguarded

exploitation of IP (Intellectual Property)

2.13 Finally, the programme, in line with the ERDF Operational Programme

(Welsh European Funding Office, 2010), is also intended to contribute

towards the overall targets for cross-cutting themes - environmental

sustainability (20% of companies to adopt or implement environmental

action plans), and equal opportunities (50% of employers adopting or

improving Equality & Diversity strategies and monitoring systems). In

achieving these objectives the SMARTCymru business plans indicate

that the programme will support companies by referring them to obtain

advice from the All-Wales Regional Support Centre (now Business

Wales centres). The programme’s outputs were also anticipated to

include environmental impacts through the introduction of new products

and processes.

2.14 Monitoring of the SMARTCymru programme is undertaken through a

suite of indicator targets. This includes both final targets, and profiled

forecasts covering the full period of the programme. The final

Convergence and Competitiveness programme targets are set out in

table 2 below.

Table 2. SMARTCymru target indicators

Indicator Convergence target Competitiveness target

Enterprises financially supported (number)

200 125

Gross jobs created (FTE) 120 75

Investment induced (GBP) £800,000 £500,000

Collaborative R&D (number) 24 15

New or improved products, processes or services launched

80 45

Products, processes or services registered

80 45

Source: WEFO projects database (Welsh European Funding Office, n.d. a; Welsh European

Funding Office, n.d. b)

2.15 The Innovation team are understood to be in discussion with WEFO to

reprofile the Convergence area targets identified in table 2, to take

account of the challenges it has faced in implementing the

SMARTCyrmu programme (see section 3 for more details of these

challenges). The proposal is for the Convergence target for enterprises

financially supported to be reduced from 200 to 120.

Programme funding

2.16 As noted in the introduction, the SMARTCymru programme is funded

through the EU Convergence and Competitiveness ERDF programmes,

with additional public funding provided by the Welsh Government, and

match funding from the private sector beneficiaries. The financial profile

intended for the programme is outlined in table 3 below:

Table 3. SMARTCymru programme funding

Convergence Competitiveness

ERDF £8,000,000 £4,980,000

Welsh Government £671,000 £303,558

Private sector match funding £9,478,157 £6,195,237

Gross spend £18,149,954 £11,478,795

Sources: (Welsh European Funding Office, 2012a; Welsh European Funding Office, 2012b)

Programme management

2.17 The SMARTCymru programmes is managed by a dedicated team within

Welsh Government’s Innovation team. It is managed by a senior director

based in the Welsh Government Penllergaer office, with overall

responsibility for the programme. The SMARTCymru team includes

technical appraisal and finance and monitoring staff.

2.18 In addition to the dedicated SMARTCymru team members the Welsh

Government’s Innovation Specialists (funded by the Business Innovation

programme) play an important role in the SMARTCymru process. They

work closely with the company to understand their innovation needs, and

support the application process. This may be for an application to

SMARTCymru, or to other support mechanisms. The role of the

Innovation Specialist is particularly important for SMEs, where there is

limited experience or capacity to plan, develop, assess and fund R&D

projects. Here, the Innovation Specialist is able to help businesses in

developing the scope of a project and determining the most appropriate

support route. They are then able to provide application support and

ongoing assistance.

2.19 Once companies submit an application to the programme a technical

appraisal is conducted by the SMARTCymru team, supported by

monitoring and due diligence staff, to ensure financial due diligence.

Final approval is made by the senior programme manager, and a formal

offer letter is sent.

2.20 When a project is underway the Monitoring team will maintain contact

with the SME in order to monitor and quantify project benefits. The

Innovation Specialists will also seek to maintain contact with the

business, with a particular focus on assisting with any future innovation

support requirements (for example, a subsequent phase of

SMARTCymru R&D).

Summary

2.21 The preceding section sets out the key elements of the SMARTCymru

programme, and its underpinning rationale. This is summarised in an

integrated (Convergence and Competitiveness) project logic model

diagram shown in figure 2 below:

Figure 2. SMARTCymru programme logic model

Inputs Activities Outputs Outcomes Impacts

ExperimentalDevelopment (ED)

Technical & Commercial Feasibility Study (TCF)

Exploitation

Convergence (gross)- £18.1 million

Competitiveness (gross)- £11.5 million

Industrial Research (IR)

New Product, processes or services

registered

Innovation impact

Economic impact

Social & environmental impact

New collaborative R&D projects

Gross jobs created

New or improvedproducts, processes or

services launched

R&D / Investmentinduced

R&D behavioural additionality

Context & Rationale

New knowledge toenable

development/improvements of products,

processes or technologies

Investigation of technical and commercial viabilityof new products, processes ortechnologies

New prototypes, conceptformulation, design ofproducts, processes,

services, initial demonstration

Marketing, publicity, advertising, sales

literature, trade fairs and product certification

Need Policy alignment

• Contributing to the Economic Renewal Plan, the Science Strategy, and Innovation Wales,

• Contributing to the objectives of the Innovation, R&D and Technology Strategic Framework

• RD&I recognised as a key driver of business growth • Comparatively low levels of business expenditure on R&D (BERD) in

Wales • Identified barriers to RD&I activity in businesses - lack of finance,

lack of new ideas, lack of time, and difficulties in appraising risk • Key barriers recognised to be strongest in SME sector

2.22 This model demonstrates a logical flow from an identified need, a set of

delivery activities and intended outputs, through to the achievement of

outcomes and impacts in both the Convergence and Competitiveness

areas. The main potential weakness in the logic model relates to the

support for social and environmental cross-cutting themes/benefits,

where the model is reliant on building strong synergies between

specialist support measures for the cross-cutting themes.

3 The programme context

3.1 The implementation of the SMARTCymru model has faced a number of

contextual challenges since its introduction in 2010/11. This has

included challenges associated with the economic recession, and

significant management issues linked to the introduction of the ERP by

the Welsh Government in 2010 (Welsh Government, 2010).

The economic and business innovation context

3.2 In relation to the economy, SMARTCymru was implemented during a

period of economic recession, followed by slow recovery. Regional GVA

is the UK’s preferred measure of economic value at the regional level.

On a ‘per head’ basis estimates produced by National Statistics (see

table 4 below) suggest that GVA in Wales, in 2012, was 72.3 percent of

the UK average, and the lowest amongst the devolved countries and

English regions. Like the other parts of the UK, Wales also suffered from

the global economic slowdown in the early part of the programme’s

introduction. This saw Wales’ GVA per head fall between 2008 and

2009, recovering in 2010 and 2011.

Table 4. Workplace based GVA at current basic prices

GVA per head (£)2

2007 2008 2009 2010 2011 2012

United Kingdom 20,643 21,026 20,472 21,023 21,368 21,295

North East 15,569 15,694 15,294 15,723 15,842 16,091

North West 17,206 17,408 16,973 17,532 17,754 18,438

Yorkshire and The Humber 16,922 16,999 16,458 16,862 17,037 17,556

East Midlands 17,820 17,887 17,355 17,832 18,083 17,488

West Midlands 17,125 17,190 16,691 17,218 17,486 17,429

East of England 19,390 19,338 18,579 19,025 19,355 19,658

London 33,744 35,046 34,910 35,422 35,638 37,232

South East 21,636 21,877 21,266 21,898 22,369 23,221

South West 18,400 18,632 18,244 18,798 19,093 19,023

England 20,681 20,992 20,531 21,054 21,349 21,937

Wales 15,097 15,179 14,794 15,407 15,696 15,401

Scotland 19,653 20,124 19,789 20,314 20,571 20,013

Northern Ireland 16,093 16,117 15,635 16,203 16,531 16,127

Source: (Office for National Statistics, 2013)

3.3 Business R&D expenditure data provides further evidence on the

context facing the SMARTCymru programme. Here the latest National

Statistics data shows that business expenditure on R&D in Wales is now

one of the lowest amongst in the UK regions and devolved

administrations. Table 5 below, for example, indicates that business

R&D expenditure in Wales was flat in the 2008-2009 period, before

declining in 2010. Despite recovery in 2011 and 2012, Wales’ business

R&D expenditure continues to lag behind other regions and devolved

administrations as a proportion of regional GVA -1.17 per cent, against

the UK average of 2.04 per cent.

Table 5. Business expenditure on R&D

Business Expenditure

on R&D (£ million)

2008 2009 2010 2011 2012

United Kingdom 15814 15532 16053 17408 17107

North East 318 315 308 259 282

North West 2130 1926 2074 2260 1784

Yorkshire and the

Humber

433 454 488 543 603

East Midlands 976 992 1137 1149 1203

West Midlands 886 847 886 1237 1461

East 4182 3812 3851 3638 3449

London 1109 907 877 1142 1477

South East 3466 3758 3798 4528 4086

South West 1345 1349 1454 1358 1364

England 14847 14360 14873 16113 15708

Wales 243 243 234 255 272

Scotland 554 631 622 689 707

Northern Ireland 171 297 324 352 420

Source: (Office for National Statistics, 2013)

The policy and organisational context

3.4 The policy and organisational context for SMARTCymru has undergone

a significant period of change in the period since its launch. Much of this

turbulence was associated with the publication and implementation of

the ERP in 2010. This document set out the Welsh Government’s

strategy for economic development, and identified its priorities for future

years. Within the ERP, ‘Encouraging Innovation’ was identified as an

important priority, with the objective of promoting the importance of

innovation (R&D, product and service innovation) for business.

3.5 While SMARTCymru is consistent with the objectives of the ERP, other

changes linked to the implementation of the new strategy had significant

implications for the programme, most notably the requirement for all

business finance (including grants) to be repayable. The implication of

this policy change was that the SMARTCymru programme was no

longer able to offer support in the anticipated format (grant). This change

resulted in a period of internal review to determine whether a repayable

grant was possible under the terms of the ERDF funding scheme, which

continued until the 2011-2012 period.

3.6 The ERP also established a number of priority sectors (six, subsequently

increased to nine). To maximise the focus on the needs of these sectors

the Welsh Government department for Business, Enterprise, Science

and Technology (BETS) was reorganised into sector teams. While the

Innovation team was positioned outside the sector teams, the Innovation

Specialists were reassigned to the sector teams.

3.7 The implications of these changes were significant. The confusion and

difficulties associated with the transition to repayable finance, for both

businesses and Welsh Government staff, resulted in the short term

growth of applications under the old grant regulations (as companies

sought to begin their projects before repayable finance was introduced),

before a substantial decline after the change was implemented in Q3

2010-11 (see figure 3). Alongside this, the reorganisation of the

Innovation Specialists meant that direct line management responsibility

moved from the Welsh Government Innovation team, to the Sector

teams. In practice, this led to the focus of the Innovation Specialists

moving from the regional to a sector focus (irrespective of geography).

The Innovation Specialist interviews conducted for this evaluation

revealed that regional stakeholder contacts were damaged in the

process.

Figure 3. SMARTCymru programme applications

Convergence project start date

Competitiveness project start date

0

5

10

15

20

25

30

35

40

45

50

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1

2009-10 2010-11 2011-12 2012-13 2013-14

RD&I Applications

Source: (Welsh Government, n.d.)

3.8 A further implication of the move towards sector teams was the transfer

of promotional responsibility to the sector teams. This hindered the

ability of the Innovation team in promoting the programme directly,

without the sector team, and is likely to have contributed to the decline in

applications in the subsequent period.

3.9 Programme branding was reviewed at the same time as the introduction

of ERP. In this respect the programme had been launched as part of the

Single Investment Fund. This, however, replaced the well-established

brand of SMART, which had been used for a number of years, and was

shared by the UK scheme operated by TSB. Interviews with the

SMARTCymru staff indicate that the lack of a recognisable brand further

contributed to confusion surrounding availability of the programme, and

the subsequent decline of applications.

3.10 The election of a new Welsh Government in 2011 marked a turning point

for the ERP inspired changes, with the subsequent announcement by

the new Minister that grants would no longer need to be ‘repayable’.

While this allowed the Innovation team to ‘re-open’ the programme to

applications, ongoing confusion within the business sector, and lack of

regional Innovation Specialist capacity meant that take-up was not as

swift as might be expected.

3.11 To address these weaknesses the programme was re-branded as

SMARTCymru (RD&I Financial Support for Business) in order to raise

awareness of its availability. The line management responsibility for the

Innovation Specialists, was also switched back to the Innovation team.

3.12 While these changes have helped to encourage greater take up,

interviews with a number of Innovation Specialists suggests that the

removal of regional focus to their work during the ERP period damaged

regional linkages and network relationships. The implication of these

factors is that levels of demand for SMARTCymru continue to be lower

than they were before the ERP process. Indeed, the Innovation team

point to ongoing limitations in awareness of SMARTCymru’s availability

in Wales (companies and potential referral bodies such as universities).

The presence of other sources of grant funding such as the Welsh

Government’s own Economic Growth Fund, and other ERDF–funded

projects such as ASTUTE (ASTUTE Wales), may have also taken some

demand away from SMARTCymru, although there is no firm evidence of

this in practice.

3.13 To counter the ongoing weakness in demand the programme team

have, in recent months, sought to raise awareness in the universities,

with the objective of encouraging referrals and greater engagement in

SMARTCymru projects.

Synergies with other programmes

3.14 The operational context facing the SMARTCymru programme includes a

range of innovation and technology programmes focusing on R&D

subsidy and support. This includes a range of Welsh Government

programmes.

3.15 The Business Innovation Programme (BIP) (Welsh Government,

2012) has direct links to SMARTCymru, via its funding of the Innovation

Specialists. These Specialists, as noted earlier, provide advice and

support to the application process, and ongoing support for the

company, as part of their wider role within the Business Innovation

Programme. The BIP also provides access to Innovation Vouchers for

university projects.

3.16 In addition to the BIP, SMARTCymru is also aligned to the A4B

programme (Welsh Government, 2012). While academics are not able

to initiate a SMARTCymru project, or be a formal partner, there is

potential for them to participate in company-led projects. In this respect

there have been attempts to strengthen the referral potential between

the two programmes.

3.17 Outside of the Welsh Government’s Innovation team, Finance Wales

(Finance Wales) is an important stakeholder for SMARTCymru. Finance

Wales have made investments in a number of companies with earlier

SMARTCymru funding (11 in total). This illustrates the synergy between

SMARTCymru’s project focus, and Finance Wales’ equity funding

(typically, a £150,000 to £500,000 first time investment) to support the

development of high growth potential businesses. Finance Wales

indicate that a SMARTCymru project can help to ‘give confidence’ with

respect to their investments, particularly where the company’s

turnover/profit growth predictions are not rapid enough to be attractive to

mainstream funders. A number of potential areas for strengthening the

link between SMARTCymru and Finance Wales’ funding were also

noted.

3.18 Other programmes providing support for R&D in Wales include the

university-led ASTUTE project. This project supports technological

development, sustainability and growth in businesses in the aerospace,

automotive and high technology sectors (in West Wales and the

Valleys). Comprising a partnership of Wales’ universities in the

Convergence area, ASTUTE provides support to manufacturing

companies to create sustainable, higher value goods and services and

bring them to a global market. This support is provided on a 50 per cent

contribution basis, although under the European Commission ‘di-

minimis’ (Welsh Government) arrangements this does not necessarily

require a direct cash contribution for small levels of assistance (under

€200,000 over a three year period). Given the potential to offer financial

support for relatively small projects in the Convergence area, there is

some potential for ASTUTE to overlap with SMARTCymru. This,

however, is likely to be greater with respect to other programmes such

as the Innovation Voucher scheme, or A4B, which have a more

research-industry focus.

Summary

3.19 The findings from this section suggest that the SMARTCymru

programme has faced significant challenges resulting from the

economic, policy and organisational context. The economic downturn,

and associated decline in business R&D expenditure contributed to

lower than expected demand for the programme. This was compounded

by the changes brought about by the implementation of the ERP, notably

the decision to move towards repayable finance in place of grants and

the reorganisation of the then BETS department into sector teams.

Together these factors prevented the Innovation team from effectively

promoting the programme, and resulted in confusion, both internally and

externally, about the availability and offer of the programme.

3.20 While the confusion surrounding the programme’s availability and offer

has subsided, with the decision to revert back to grant funding, (re)using

the SMART branding, and relocating the Innovation Specialists (back) to

within the Innovation team, the challenge for the programme continues

to be one of stimulating demand in the remaining part of the programme.

The links to potential referral programmes and organisations indicate

one of the ways in which this challenge has been addressed.

4 The programme profile

4.1 The Welsh Government’s monitoring data provides an overview of

programme participation, and highlights the nature of demand at the

mid-term stage. The following section sets out the mid-term profile for

both the Convergence and Competitiveness elements, to the end of

June, 2013, based on SMARTCymru monitoring data (Welsh

Government, n.d.).

Profile of company participation

4.2 At the mid-term point there were 84 (53 complete and 31 in progress)

projects in the Convergence area, and 32 (15 complete and 17 in

progress) in the Competitiveness area. This equates to an overall total of

116 projects at the mid-term point.

Geographic location

4.3 Figures 4 and 5, below, illustrate that the 116 projects are spread across

Wales. The most prominent location for SMARTCymru projects in the

Convergence area (figure 4), by some distance, is the Swansea area

with more than double (23) the number of any other Convergence area,

and accounting for over 27 per cent of the total. A number of other

Convergence locations had between five and 11 projects, including

Bridgend (11), Denbighshire (10), Torfaen (eight), Rhondda Cynon Taf

(seven) and Gwynedd (six).

Figure 4. Geographic location of projects – Convergence area

4.4 In the competitiveness area, as might be expected, SMARTCymru

projects were primarily located in the Cardiff area (see figure 5). Indeed,

at the mid-term point half (16 of 32) of the Competitiveness projects

were based in Cardiff. This was followed by Monmouthshire and Powys

(four each). Three or less projects were located in the remaining areas.

Figure 5. Geographic location of projects – Competitiveness area

Equality and diversity

4.5 Equality and diversity data for the SMARTCymru programme is held by

the Business Innovation Programme (which manages the Innovation

Specialists), and was not available at the time of the mid-term

evaluation. Discussions with the manager of the Business Innovation

Programme, however, suggest that the vast majority of applicants to the

Business Innovation programme fall within a narrow equality and

diversity profile characterised as ‘white, middle aged, males’.

Company size

4.6 In the Convergence area figure 6 below indicates that the majority of

company participants, at the mid-term stage, are relatively small,

reporting turnover of less than £1 million (67 per cent of projects). Within

this category the largest single group of projects are companies with a

turnover of less than £100,000, accounting for 23 per cent of projects.

Figure 6. Participant company turnover – Convergence area

4.7 Figure 7, in contrast, reports on turnover in Competitiveness area

participants. This points to the comparatively larger size of companies in

Competitiveness area projects, with a quarter reporting a turnover

greater than £10 million. This figure is consistent with the relative

prosperity of the Competitiveness area. The turnover data does,

however, also point to strong participation in smaller companies, with the

majority of projects (nearly 67 per cent of projects or 16 of 24) reporting

less than £1 million turnover.

Figure 7. Participant company turnover – Competitiveness area

4.8 The employment profile of the SMARTCyrmu projects provides a further

indication of company size. Here, figure 8 suggests that some 67 per

cent of Convergence company projects, employ between 1 and 10

employees.

Figure 8. Employees – Convergence area

4.9 Similarly the largest proportion of Competitiveness area participants

employ 1-10 employees (68 per cent of projects). However, three

Competitiveness area projects involved companies with more than 250

employees (figure 9).

Figure 9. Employees – Competitiveness area

4.10 Taken together the employment and turnover data confirms that the

majority of companies participating in SMARTCymru can be classified as

SMEs3, with only three projects falling into the large sized firm category.

Industry sectors

4.11 The profile of participation is further illustrated by the industry sectors of

SMARTCymru companies. Figure 10, for example, suggests that the

Advanced Manufacturing and Materials sector accounts for the majority

of projects in the Convergence area. This is closely followed by ICT

sector projects (28 per cent). Financial and Professional Services and

Life Sciences are also represented (29 per cent), alongside two

companies from Energy and the Environment.

Figure 10. Sectors – Convergence area

3 According to European Commission definitions (European Commission)

4.12 The importance of the Advanced Manufacturing and Materials and the

ICT sector is also evident in Competitiveness area projects (66 per

cent). Life sciences projects are also strongly represented in the

Competitiveness area projects (34 per cent). In overall terms, however,

a smaller range of sectors are represented in the Competitiveness area

projects.

Figure 11. Sectors – Competitiveness area

RD&I Phase

4.13 The SMARTCymru programme funds projects according to RD&I

phases, with companies expected to make their way through the phases

in a sequential manner. At the mid-term stage, for example, it could be

anticipated that the balance of participation is towards the earlier RD&I

phases such as TCF and IR. Figure 12 confirms that this is the case,

with 70 per cent of projects in the TCF and IR phase. A smaller

proportion (30 per cent) are currently in the Experimental Development

Phase (25 per cent) and Exploitation phase (5 per cent). This profile, will

continue to evolve throughout the remaining period of the programme.

Figure 12. RD&I Phase – Convergence area

4.14 Figure 13 provides phase data for the Competitiveness area. Here a

larger proportion of projects fall within the early phases (TCF and IR

account for 31 per cent and 35 per cent respectively). This, however,

can be explained by the later start date for the SMARTCymru

Competitiveness programme. Only 6 per cent of projects have

reached/completed the Exploitation phase.

Figure 13. RD&I Phase – Competitiveness area

Programme spend

4.15 The Convergence programme costs at the mid-term stage suggest that

31 per cent of the budget has been used, with income received at a

similar level. As table 6 shows, both expenditure and income figures are

lower than their profiled/forecasted targets - 69 per cent and 68 per cent

respectively. Each expenditure category for the Convergence

programme is lower than forecasted with administration and legal and

professional fees at 17 per cent and 24 per cent of their forecasts

respectively. Only 68 per cent of the forecasted grants has been spent

while the only expenditure close to its forecast is staff costs at 97 per

cent.

4.16 In terms of income, private match funding is well below its forecast at 64

per cent, while 92 per cent of the Welsh Government funding forecast

has been received. This reflects the higher than expected number of

projects in the early SMARTCymru phases (TCF and IR projects), where

there are higher intervention rates. This pattern of projects has resulted

in the Welsh Government funding being higher (and vice versa for

private match funding) than would have been the case had more

projects made their way to the later SMARTCyrmu phases.

Table 6. Mid-term programme cost – Convergence area

Convergence

80504 EXPENDITURE TO JUNE 2013

Expenditure Type

Cumulative

Expenditure To Date

Eligible (GBP)

Delivery Profile

Cumulative Expenditure

To Date (Eligible GBP)

% of Delivery Profile

Cumulative Expenditure

To Date spent

Total Forecast

Project Cost

Eligible (GBP)

Forecasted budget

remaining

% of budget

used

Administration 105£ 624£ 16.83% £ 1,308 £ 1,203 8.03%

Grants 5,160,819£ 7,617,361£ 67.75% £ 17,385,528 £ 12,224,709 29.68%

Legal & Professional 16,637£ 68,122£ 24.42% £ 131,252 £ 114,615 12.68%

Staff 376,366£ 389,838£ 96.54% £ 539,359 £ 162,993 69.78%

Travel & Transport 8,017£ 11,593£ 69.15% £ 18,718 £ 10,701 42.83%

Total 5,561,945£ 8,087,538£ 68.77% £ 18,076,165 £ 12,514,220 30.77%

80504 INCOME TO JUNE 2013

Organisation

Cumulative Income To

Date

Delivery Profile

Cumulative Income To

Date (Eligible GBP)

% of Delivery Profile

Cumulative Income To

Date received

Total Forecast

Project Income

Total Forecast

Project Income

remaining

% of Total

Forecast Project

Income received Various Private Match Funders 2,502,222£ £ 3,907,978 64.03% £ 9,422,915 6,920,693£ 26.55% WAG The Economic Development

Group within DEandT 604,034£ 660,029£ 91.52% £ 653,259 49,225£ 92.46%

Total 3,106,256£ 4,568,007£ 68.00% 10,076,174£ 6,969,918£ 30.83%

Source: (Welsh Government, n.d.)

4.17 For the Competitiveness programme, table 7 shows that activity and

expenditures have been far lower than forecasted, even when compared

to the Convergence programme. The total percentage of the forecasted

budget used and income received is close to only 8.5 per cent of the

profiled target. Only 22 per cent of the forecasted grants expenditure at

the mid-term stage has been used, and every other expenditure

category is between 9 per cent and 26 per cent of the forecast. In

contrast to this and similar to Convergence, staff expenses have

exceeded its forecast by 8 per cent; with £99,024 spent compared to the

forecast of £91,456.

4.18 Income received from Competitiveness projects to date is only 24 per

cent of its forecast to June 2013 with income received from private

match funders as low as 17 per cent of that profiled. Income received

from Welsh Government, however, is slightly more than had been

forecast with £195,070 received compared to the forecast of £194,537.

This, again, is a result of the higher number of projects than expected

that engaged in the early phases where there are higher intervention

rates.

4.19 Taken together these figures highlight the contextual challenges faced

by the programme in generating and maintaining demand, and are

consistent with numbers of companies supported.

Table 7. Mid-term programme cost – Competitiveness area

Competitiveness

80504 EXPENDITURE TO JUNE 2013

Expenditure Type

Cumulative

Expenditure To

Date (Eligible GBP)

Delivery Profile

Cumulative

Expenditure To Date

% of Delivery Profile

Cumulative Expenditure

To Date spent

Total Forecast

Project Cost

(Eligible GBP)

Forecasted

budget

remaining

% of budget used

Administration £ 37 £ 432 8.56% £ 888 £ 851 4.17%

Grants £ 855,718 £ 3,865,475 22.14% £ 11,115,267 £ 10,259,549 7.70%

Legal & Professional £ 12,360 £ 47,650 25.94% £ 101,584 £ 89,224 12.17%

Staff £ 99,024 £ 91,456 108.28% £ 191,140 £ 92,116 51.81%

Travel & Transport £ 1,240 11,593 £ 10.70% £ 9,228 £ 7,988 13.44%

Total 968,380£ 8,118,155 £ 11.93% £ 11,418,107 £ 10,449,727 8.48%

80504 INCOME TO JUNE 2013

Organisation

Cumulative

Income To Date

(Eligible GBP)

Delivery Profile

Cumulative Income

To Date

% of Delivery Profile

Cumulative Income To

Date received

Total Forecast

Project Income

(Eligible GBP)

Total Forecast

Project Income

remaining

% of Total

Forecast Project

Income received

Various Private Match Funders 353,185£ 2,075,475 £ 17.02% 6,176,612£ £5,823,427 5.72%

WAG The Economic Development

Group within DEandT 195,070£ 194,537 £ 100.27% 261,495£ £66,425 74.60%

Total 548,255£ 2,270,012 £ 24.15% 6,438,107£ £5,889,852 8.52%

Source: (Welsh Government, n.d.)

Summary

4.20 The results of this section suggest that SMARTCymru programme

(Convergence and Competitiveness areas) participants, at the mid-term,

have been largely micro and small companies, according to the

European Commission definitions. This is consistent with the target

group for the project (SMEs). The participants are primarily located in

Wales’ major urban areas on the M4 corridor, with Cardiff and Swansea

accounting for a large proportion of company participants (36 per cent).

Sector participation is consistent with the Welsh Government’s sector

priorities, with the Advanced Manufacturing and Materials and ICT

sectors accounting for the largest proportion of projects (74 per cent in

the Convergence and Competitiveness areas). Finally, the mid-term

profile suggests that the most projects are in the earlier RD&I phases of

their projects (68 per cent are in TCF and IR phases while only 5 per

cent reached/completed the Exploitation phase).

4.21 Programme costs are below the forecasted targets at the mid-term. This

can largely be explained by the difficulties faced in adapting to the policy

and organisational changes associated with the ERP process, on one

hand, and the economic downturn/recovery on the other.

4.22 Programme costs are below what was expected at the start of the

project, with 31 per cent of the Convergence budget used, against just

over 8 per cent of the Competitiveness budget. Within this, expenditure

on grants and private sector match funding have also been significantly

below target in both the Convergence and Competitiveness areas. Of

particular concern, however, is the comparatively high levels of Welsh

Government match funding. In the Convergence area, for example, the

level of Welsh Government match funding is consistent with the profiled

target (i.e. 100 per cent), in contrast to the private match contribution

which stands at 17 per cent. This imbalance between Welsh

Government and private sector match funding is linked to the higher

than expected number of projects in the early phases of the programme,

where intervention rates are higher (meaning that Welsh Government

match funding is higher). While the overall private and Welsh

Government match funding income is likely to balance out to some

degree as companies move through the programme (to the later phases

with lower intervention rates), there remains a risk that Welsh

Government match funding will be exhausted. This highlights the key

challenge for the programme in encouraging companies to progress to

Experimental Development RD&I phase.

5 Outputs and impacts

5.1 This section of the report summarises the key impacts arising from a

small sample of SMARTCymru projects; contrasting these with the

outputs indicators collected by the programme team. The 10 companies

interviewed for the evaluation accounted for 17 projects (equivalent to 15

per cent of the total number of projects based on those completing one

or more stage). Seven out of the 10 interviewees were based in the

Convergence Region.

SMARTCymru output indicators

5.2 Tables 8 and 9, below, illustrate the primary outputs and results

collected for the SMARTCyrmu programme against the mid-term

business plan targets. Based on the column ‘Cumulative claims to 2013

Q2’4 this data suggests that the programme is some way behind the

original forecasted profile targets. Results for Convergence exceed

those of the Competitiveness area, with 85 per cent of the profiled

targets for gross jobs created and 61 per cent of the investment induced

profile target (to date) achieved. The number of (Convergence)

enterprises financially supported is, however, some way below the

profiled targets, with only 27 per cent of the delivery profile target

achieved to date). It should be noted, however, that the profiled targets

reflect assumptions made at the start of the process, and do not take

account of the decline in applications at reflects the decline in

applications brought about by the ERP process.

5.3 As for other indicators, there are results reported for collaborative R&D

projects (10), products, processes or services registered (10), and new

or improved products, processes or services launched (39), but delivery

profile target to date figures weren’t provided for these.

5.4 As for the Competitiveness area, generally the achievement of profile

target to date of indicators is far lower. The only indicator with a

significant proportion of profile target achieved is collaborative R&D

4 SMARTCymru monitoring data is collected during the financial claims process. The

‘Cumulative Claims to 2013 Q2’ refers to all results and outputs data collected by the end of June 2013. This data is presented in its unverified format, and may change once the SMARTCymru monitoring process is complete.

projects, with two thirds (4 of 6) of its profile target to date achieved. No

other indicator has reached half the profile target to date, with new or

improved products, processes or services launched, next in line at 23

per cent. The achievements for Gross jobs created and enterprises

financially supported are relatively weak, with only 13 per cent and 14

per cent of profile target to date achieved respectively. It should be

noted, however, that the Competitiveness programme projects have had

less time to build up such outputs.

Table 8. Convergence output indicators

Indicator Indicator Type

Cumulative claims to 2013 Q2

Delivery Profile Target To Date

Achievement of profile target to date (%)

Project Final Target

Remaining to Final Target

% Current Final Target Achieved to date

Enterprises Financially Supported

Output 25 92 27% 200 -175 13%

Gross Jobs Created

Result 29 34 85% 120 -91 24%

Investment Induced Total

Result £229,404 £374,149 61% £800,000 £-570,596 29%

Collaborative R&D Projects

Output 10 n/a n/a 24 -14 42%

Products Processes or services registered

Result 10 n/a n/a 80 -70 13%

New or improved products, processes or services launched

Result 39 n/a n/a 80 -41 49%

Source: (Welsh Government, n.d.)

Table 9. Competitiveness output indicators

Indicator Indicator Type

Cumulative claims to 2013 Q2

Delivery Profile Target To Date

Achieveme nt of profile target to date (%)

Project Final Target

Remaining to Final Target

% Current Final Target Achieved to date

Enterprises Financially Supported

Output 7 49 14% 125 -118 6%

Gross Jobs Created

Result 2 15 13% 75 -73 3%

Investment Induced Total

Result £34,721 £160,000 22% £500,000 £-465,279 7%

Collaborative R&D Projects

Output 4 6 67% 15 -11 27%

Products Processes or services registered

Result 2 13 15% 45 -43 4%

New or improved products, processes or services launched

Result 3 13 23% 45 -42 7%

Source: (Welsh Government, n.d.)

5.5 Interpretation of the output indicators is complicated by the process that

has been adopted for their collection. This relies on the company

participants to provide such data alongside their financial claims. Given

that some companies have yet to complete a financial claim, it is likely

that the figures underestimate the actual achievements at the mid-term.

It should further be noted that the collection and processing relies on

Welsh Government’s own staff resources. Interpretation is also

complicated by the long-term nature of the innovation process, and the

likelihood that outputs can take up to four years to deliver and build,

according to estimates (PricewaterhouseCoopers, 2009).

5.6 Indicators for cross-cutting theme outputs are collected by the Innovation

Specialists, and recorded against the Business Innovation Programme.

Difficulties in sharing this information across the programmes do,

however, exist (linked to the anonymity of certain datasets), and suggest

an area where further investigation will be required for the final

evaluation.

Business interview results

Baseline R&D activity

5.7 Prior to engaging with SMARTCymru the majority of companies were

active in conducting R&D (six out of 10 interviewed). The remaining

companies reported no previous activity, but this can largely be

expected due to small size of the participants. A number of companies

were created explicitly as part of the SMARTCymru process (three out of

10).

5.8 Of the six companies that reported prior R&D activity, four out of six had

a dedicated R&D budget, and five out of six had dedicated R&D

employees. The average R&D budget for the four companies who

reported one was £195,000 and the average dedicated number of R&D

employees for the five companies who had one was four.

5.9 These results suggest that a high proportion of interviewed companies

were ‘R&D experienced’, with dedicated R&D budgets and employees.

However, the results also indicate that companies’ R&D activities were

undertaken internally with limited experience of other external R&D

support. Indeed, only one out of ten cited previous public sector R&D

support and only two out of 10 citing previous engagement in

collaborative R&D projects.

Project significance

5.10 The interview results further suggest that projects were strategically

significant to the company, with six of the 10 companies indicating that

they probably would not have survived had it not been for the

programme, with a further two pointing to the significance of the project

for future turnover (see table 10). As noted earlier, three companies

were created explicitly as part of the SMARTCymru process. Two more

reported that without SMARTCymru support they wouldn’t have

recorded any turnover or would have struggled for a period before going

out of business. All companies indicated that their project was significant

to its future prospects. These results underpin the importance of

SMARTCymru support in developing or in addressing important strategic

company needs.

Table 10. How important is the outcome of the project for the company?

Frequency % The outcome is crucially important to the company, without which we probably wouldn’t have survived 6 60 The project outcome is very important, it will account for a significant proportion of turnover in the near future 2 20 The project outcome is important to the firm but we have a number of other products or services in the pipeline 2 20 The outcome of this project is of modest importance to the company 0 0 The outcome of this project is not important to the company 0 0

Project achievements

5.11 The findings from the interviews suggest that project objectives have

been met by a moderate to very large extent (see table 11 below), with

the majority of these falling into the large / very large extent categories

(nine out of 10). One company reported that the project met and

exceeded its objectives to date; ‘we are further ahead than we thought’

while another reported that the project was a ‘great success with all

objectives met’. Indeed, this company had managed to introduce its

products into some of the largest financial services companies in the UK.

Table 11. To what extent were the project objectives met?

Frequency %

Very large extent 5 50

Large extent 4 40

Moderate extent 1 10

Small extent 0 0

No extent 0 0

N/A 0 0

5.12 This positive picture of the programme is further highlighted in table 12

below where all of the companies indicated strong prospects for

commercialisation. Indeed nine out of 10 companies noted that their

project had the potential to bring some / significant benefits. The

remaining company also expected to receive further benefit but it was

too early to say. One of the interviewees, for example, reported that his

company had already managed to commercialise the results of an

earlier SMARTCyrmu project, and were currently working on a second

project to add to their offer. The tendency for optimism bias (HM

Treasury, 2013) in such assessments should, however, be noted.

Table 12. Prospect of commercialisation

Frequency %

Significant benefits/potential 8 80

Some benefits/potential 1 10 Company only exploitation 0 0 Expected but too early 1 10

No possibility 0 0

N/A 0 0

Protection of results captured

5.13 Results of the SMARTCymru projects were, in many cases, protected

through formal mechanisms including patents, copy rights, trade marks

etc. Table 13, below, suggests that six of 10 companies had some form

of protection (ranging from limited to complete) for their project outputs.

A number of the companies indicated that formal protection mechanisms

were not necessary or available because the

products/processes/services developed were entirely based on know-

how and therefore not protectable (for example, patent searches

revealed they were free to operate in this area). In this respect one

company estimated that it would take up to one year to copy the

company's product. Another company indicated that their strategy for

protection was to keep innovating the service in order to make it difficult

for competitors to copy or to provide similar quality. This respondent

reported that the Welsh Government can be too ready to prioritise

patents, even in cases where they were not perceived to be relevant

(e.g. ICT service projects).

Table 13. Protection of IP

Frequency %

Completely protected 3 30 Limited protection 3 30 No protection 4 40

Satisfaction with SMARTCymru

5.14 The results of the interviews suggest a positive picture in relation to

satisfaction with the support provided by SMARTCymru (see table 14).

On a scale of one to five, where one equated highly unsatisfied and five

highly satisfied, the average response from the ten companies was four

point three. In fact, four out of 10 companies were very satisfied and half

(five of 10) were satisfied with Welsh Government's support in

developing their project, with the remaining company moderately

satisfied.

Table 14. Satisfaction with Welsh Government's support in developing projects

Frequency %

Very satisfied 4 40

Satisfied 5 50

Moderately satisfied 1 10

Unsatisfied 0 0

Very unsatisfied 0 0

5.15 The application process was described as being ‘long-winded’ by a

number respondents with the Welsh Government team felt to lack an

understanding of the ‘commercial urgency’ associated with innovation

projects. Companies would often have tight timetables for launch and

any delays were said to be potentially ‘catastrophic.’ One company

interviewed claimed it was close to being wound up as a consequence of

this process.

5.16 One of the larger, more established companies, suggested that the

process should become ‘more streamlined’. The company felt that

undertaking a TCF was unnecessary because the basic research had

been completed internally5. The general consensus was that the process

should become more flexible with the exploitation and commercialisation

phase introduced earlier if appropriate.

5.17 Another company was very critical of the exploitation phase and had

decided not to press ahead with it after having initial discussions with the

SMARTCymru team. The issue here was that the company viewed a

marketing approach focussed on ‘selling the company’ as the best route

ahead rather than focussing solely on marketing the product itself. The

company claimed to have had backing from three independent Welsh

PR companies on this strategy but was rejected by SMARTCymru for

being ‘too broad’.

Impacts analysis

It should be noted, however, that companies are not required to conduct at TCF, if they can already prove that they have undertaken such activity prior to their SMARTCymru application.

5

5.18 The following section seeks to outline the benefits of the SMARTCymru

projects to Wales at the mid-term stage. The analysis takes into account

both quantitative and qualitative analyses to provide an overall picture of

the impact of the SMARTCymru programme.

Project additionality

5.19 The interviewed companies were asked about the extent to which

projects would have progressed without SMARTCymru funding. Here

the results suggest that none of the projects would have gone ahead

unchanged in terms of timescales, scale or location, had it not been for

SMARTCymru funding. This represents significant additionality in

SMARTCymru funded projects, and demonstrates the relative

importance placed on the funding programme by the companies

interviewed.

5.20 In examining the responses to this question in more detail, three of 10

respondents argued that their project would not have taken place at all,

had it not been for SMARTCymru funding. This assessment was not

altogether hypothetical for one of the companies interviewed, with the

respondent reporting that they did abandon their project once the

funding basis changed to a repayable grant. The project did, however,

continue when the Welsh Government later contacted the company to

offer grant funding terms. This reflects the higher degree of uncertainty

and risk associated with industrial research projects (contrasted with

potential higher returns), in comparison with near market R&D.

5.21 For projects that would have gone ahead, the perception of half (five of

10) the respondents was that without the SMARTCymru funding the

projects would have taken longer to complete. Other respondents

predicted that their project would have gone ahead on a reduced scale

(one of 10) or at a later date (one of 10). The project outcomes therefore

would have been delayed and the benefits potentially reduced for the

vast majority (seven of 10) of projects (see table 15).

Table 15. Action taken in projects had SMARTCymru funding not been

available

Frequency % Gone ahead as now unchanged – that is the same scale, time 0 0

and location

Gone ahead at a later date (a delay in project) – on a same scale 1 10 Gone ahead but would have taken longer to complete 5 50 Gone ahead but on a reduced scale 1 10 Gone ahead but lower quality 0 0 Been abandoned 3 30

5.22 Finally, had SMARTCymru funding not been available companies

indicated that other funding sources may have been considered (see

table 16), including bank loan (23 per cent), equity finance (15 per cent),

own capital (15 per cent), other public sources (8 per cent), and bank

overdraft (15 per cent). Three companies, however, reported that they

would not have gone ahead with the project, had funding been

unavailable. For many of the companies who suggested that they would

have looked for other options, they didn’t consider any to to be

particularly viable in reality. One of the larger companies interviewed

stated that it would have been very difficult to make a case for funding

internally, and if SMARTCymru assistance hadn't been available, the

team would have had to consider whether it was possible to proceed; ‘it

would have been touch and go’. This was consistent with answers

reported to the earlier question.

Table 16. Other sources of funding sought had SMARTCymru not been

available

Frequency %

Bank loan 3 23 Bank overdraft 2 15

Equity finance 2 15

Other public sector organisation 1 8

Own capital 2 15 Joint venture 0 0 None (project would not have gone ahead) 3 23

Other 0 0

Innovation impacts

5.23 A key indicator of benefit to a region in supporting RD&I programmes is

the development and launch of new/improved products; processes;

and services by companies participating in the programme. This factor

represent key objectives of funding R&D.

5.24 Table 17 below shows, the interviewed companies produced 10 new

products, six of which represented entirely new products to the market.

One new service was developed to the company, market and Wales,

and there are also four improved products in Wales from companies in

CMI’s sample due to SMARTCymru support.

5.25 These results indicate that in funding innovative product and process

development, SMARTCymru is supporting projects that are consistent

with its objectives. They also highlight the challenges for the programme

in generating demand from companies with service and process

projects.

Table 17. New products/process/services as a result of SMARTCymru

Company Wales Market

New products 10 10 6

New processes 0 0 0

New services 1 1 1

Improved products 0 0 4

Improved processes 0 0 0

Improved services 0 0 0

5.26 Encouraging additional R&D expenditure is a complementary

objectives (to the development of products, processes and services),

and can help to embed R&D activity within a company. The intention,

here, is that companies will continue to undertake R&D over the longer

term, with potential spin-off benefits for Wales. The results from the

interviews suggest that capacity has been built in key areas, most

notably in the ability of companies to invest more in R&D as a result of

SMARTCymru. This is supported by the interview evidence (see table

18), where nine of 10 companies indicated that they now invest more in

R&D as a result of the support received. Of the eight companies that

provided an estimated figure for this, the average investment made as a

result of SMARTCymru was £282,750 per annum.

Table 18. Have you invested more in R&D as a result of the project?

Frequency %

Yes 9 90

No 1 10

5.27 Leverage of additional funding is an area where further R&D funds can

be raised. Here, the interviews pointed to examples where companies

had participated in subsequent R&D projects funded through sources

such as SMARTCymru, FP7 and TSB. The findings also pointed to

improved capability to apply for such funding, with half (five of 10)

reporting that SMARTCymru has encouraged/improved their capability

to apply for competitive R&D funds to some extent (see table 19). These

benefits illustrate SMARTCymru’s role in generating so-called R&D

behavioural additionality in supported companies.

Table 19. Extent to which SMARTCymru encouraged/improved capability to apply for competitive R&D funds in companies

Frequency %

To a great extent 3 30.0

To some extent 1 10.0

To a small extent 1 10.0

No impact 4 40.0

Don't know 1 10.0

5.28 New areas of knowledge and skills are important wider benefit to

participation in R&D programmes. Here the results of the interviews

suggest that SMARTCymru projects enabled company participants to

develop new technical skills (seven of 10) in areas related to the

research undertaken during the project. Related to these benefits

projects also provided the opportunity for companies to attract new skills

and improve the skills of staff (six of 10 and seven of 10 respectively). In

a small number of cases (two of 10) qualifications were also developed

through the projects. These skills and knowledge benefits were

important to the companies concerned, but also potentially to the sector

based within (and outside) Wales if, for example, staff were to leave to

take up opportunities elsewhere.

5.29 Collaborative innovation linkages are a further important area of

impact from the SMARTCymru projects. These linkages have the

potential to produce long term collaborative benefits in the areas of

research and innovation. The results from the interviews suggest that

four of 10 companies established collaborative linkages during their

SMARTCymru projects. This included relationships with the university

partners (three of 10), as well as with other companies; large companies

and SMEs (one of 10). These findings highlight that while collaboration

with other partners is not a formal requirement of SMARTCymru many of

the projects supported do include such linkages, many of which are

sustained post project.

Table 20. Maintenance of linkages and collaborative relationships post-

SMARTCymru

Frequency %

Large companies 3 25

SMEs 2 17

University/HEIs 7 58

5.30 Attitudes towards further collaboration are another area in which

SMARTCymru has helped build the capacity of companies and

universities to conduct (collaborative) industrial research. However,

results from CMI’s sample (as illustrated in table 21) suggest that this

has been a modest impact with only one company reporting a positive

impact in their attitude towards collaboration with large companies and

SMEs; and three reporting on a positive impact on their collaboration

with HEIs. The experience of one SMARTCymru project led to a

negative impact on the company’s attitude towards collaboration with

HEIs. The majority of companies, however, reported no impact – six in

terms of their attitude towards collaboration with HEIs and nine of 10

with large companies and SMEs. This, in the majority of cases was a

result of the projects containing no HEI involvement, or having pre-

existing relationships (which did not change attitudes etc.).

Table 21. What impact has the SMARTCymru programme had on your company's attitude to collaboration with:

Significant

+ve impact +ve impact No impact -ve impact

Significant

-ve impact

Large companies 1 0 9 0 0

SMEs 1 0 9 0 0

University/HEIs 1 2 6 1 0

5.31 Innovation spillover benefits were examined from a number of

perspectives in the interviews, with a focus on understanding the nature

and extent of innovation impacts beyond the immediate confines of the

companies concerned. At the mid-term stage the findings suggest that in

most cases it will be too early for spillovers to have occurred. A number

of instances of spillovers were identified however, with one company

that that they had acquired new technical knowledge through a

SMARTCymru project transferring this to a sister company, which had

subsequently gone on to develop a new area of business based on

digital interoperability services. In the context of this research it was not

possible to produce a definitive benefit statement. The collaborative

nature of many of the projects established, however, implies that this is

an area where future benefits are possible. The challenge, however, will

be to allow sufficient resources to ‘trace’ these spillovers beyond the

immediate boundary of the SMARTCymru participants.

5.32 These results suggest that collaborative linkages and positive attitudes

towards their importance are a key area of R&D behavioural

additionality. These relate to network benefits, and illustrate how the

provision of SMARTCymru funding can help to produce a longer lasting

benefit, beyond the immediate project.

5.33 Cross-cutting theme benefits were also explored in the company

interviews, with interviewed companies asked about their use of Welsh

Government support for equality and diversity and environmental

sustainability. No company, however, was able to identify either support

provided or benefits in these areas.

Economic impacts

5.34 A key aim of SMART Cymru is to produce economic impacts for Wales,

as expressed in uplifts of additional employment (jobs) and GVA

(growth) achieved as a result of the programme.

5.35 This analysis draws on the impacts reported in the WEFO monitoring

returns for June 2013. These impacts have then been adjusted for a

series of factors to allow a calculation to be made of net additional

impacts achieved by the SMARTCymru programme in the period.

5.36 The following diagram outlines, in brief, the process for calculating net

additional impact from gross economic impact results:

Assumptions

5.37 The above formula uses various elements to inform the discounting of

gross direct effects. These are outlined below.

5.38 Deadweight is a measure of how different the company perceived their

employment would have been without the SMARTCymru programme.

The results of the interviews suggested that employment would have

been 70 per cent lower had it not been for the programme. On this basis

a deadweight assumption of 30 per cent is used. This, as table 22 below

illustrates, suggests a low level of deadweight when compared to other

similar programmes in the UK. It is likely to reflect the small size of

participant companies, and the high strategic importance attached to the

SMARTCymru projects. This is a positive factor and confirms that the

Welsh Government is funding projects that are strongly additional. It also

points to likely underinvestment in such R&D projects if Welsh

Government chooses to exit from the provision of such support at the

end of the current programme.

Table 22. Comparative deadweight assessments

Name Programme type Estimated deadweight

Period

SMARTCymru RD&I 30% 2010/11 to 2013 Enterprise Ireland RTI RD&I 35% 2002-2006 IDA Ireland R&D Grant RD&I 63% 2002-2006 Invest NI START Industrial research 92% 1996-2007 Invest NI COMPETE Experimental

development 79% 2002-2006

TSB Collaborative R&D Collaborative R&D 35% 2004-2011 Sources: (Frontline Consultants, n.d.; Frontline Consultants, n.d.; CM International, 2009a;

CM International, 2009b; CM International, 2011; PACEC, 2011)

5.39 Displacement (Dp) is the proportion of benefits that are displaced

outside of the company/Wales through support from SMARTCymru. This

may occur where a new product, for example, could make that of a

Wales-based competitor obsolete. No evidence of this was found from

the company interviews, and this was consistent with most projects

being in their early stages of development. A zero percent figure has

therefore been applied to this factor, in line with current additionality

guidance (English Partnerships, 2008), and eliminated from the result

below.

5.40 Leakage (L) is the proportion of benefits achieved that have leaked

outside of Wales. The results of the interviews suggested that this was

low, principally because of the low level of job creation and the location

of companies. A 10 per cent figure has been applied to this factor in line

with additionality guidance (English Partnerships, 2008).

5.41 There is less scope to benchmark substitution (S). No evidence of

substitution was found in any of the beneficiary companies and it has

been assumed to be zero percent and eliminated from the result below.

5.42 The fact that our assessment of leakage and substitution are low may be

regarded as a positive finding that suggest benefits are not being lost to

Wales and that companies are not taking public sector resources to do

activity they would have anyway done themselves.

5.43 Finally, multipliers (M) take into account the level of usage by each

company of suppliers in Wales. CMI uses the additionality guidance

prepared by English Partnerships (English Partnerships, 2008) to

calculate the level of multiplier and has applied a composite multiplier of

1.5 to the additionality calculation representing a medium level of local

benefits anticipated to the wider supply chain.

Conversion of outputs from gross to net

5.44 Gross employment impacts of 31 jobs were created across the

Convergence and Competitiveness programme by the SMARTCymru

programme. Applying the conversion factors produces an estimate of a

net additional employment impact of 29 jobs, with an average of 0.25 net

additional jobs per project.

5.45 Gross Value Added impact for the SMARTCymru programme, at the

mid-term point, is £1.7m based on jobs created since the start of the

programme (i.e. the Actual6 GVA benefit).

5.46 Taking account of discount factors and estimated persistence of

impacts, the Cumulative GVA benefit to Wales of mid-term impacts

produces an estimated net additional GVA of £4.5m.

5.47 Future Potential GVA benefit from SMARTCymru programme, taking

account the economic impact of the programme is likely to produce an

annual £3.4m in net additional GVA, based on the estimated gross job

outputs in 2015.

6 Further details on the calculation of Actual, Cumulative and Future Potential GVA can be found in the annex.

5.48 While economic impact measured according to conventional method

such as above is low, it is important to view these results in context. A

programme such as SMARTCymru was always likely to yield limited

short-term economic benefits for a number of reasons:

5.49 Firstly, the main aim of the programme has not been to create jobs but

to increase R&D activity, and develop new products, processes and

services. These are factors where the interviews suggest that

companies are investing in further R&D and produce new product,

processes and services.

5.50 Secondly, to accrue benefits via the technology development process in

small companies takes a significant time. In this respect it should be

noted that many of the benefits (for example, jobs) associated with

SMARTCymru projects are likely to be created on the basis of

successful commercialisation of a new product, process or service.

5.51 Thirdly, the challenges associated with the economic downturn and the

Welsh Government’s reorganisation and shift away from grants

(associated with the ERP process) has impacted on demand for the

programme. These factors are likely to have contributed to the relatively

low economic impact figures at this stage.

Summary

5.52 The findings from section 6 suggest the following outputs and impacts

from SMARTCymru at the mid-term:

Development of new and improved products and process, with a high

proportion of these (six out of 10) representing new to the market

innovations

Nine of the 10 companies have invested more in R&D as a result of the

project (an average of £282,750 per business), indicating strong levels

of R&D behavioural additionality

Improved capability to access additional R&D funds (five of the 10

companies), with success in accessing funding from other public

funding sources, including FP7 and TSB (five of the 10 interviewees)

Development of new technology and market knowledge and skills of

staff (seven out of 10 companies)

Maintenance of collaborative innovation linkages (four out of 10

companies), including universities and other companies

5.53 Innovation spillovers represent a potentially important area of innovation

impact, particularly in those projects where there is a collaborative

element. A number of such impacts were identified, with the potential for

these to produce future spillover benefits at the final evaluation stage.

5.54 The economic impact results suggest that the programme results to date

will contribute 29 net additional jobs and a cumulative GVA of £4.5m.

This result has been achieved with comparatively low levels of

deadweight, again suggesting that project results are strongly additional.

It should be noted, however, that the main aim of the programme has

not been to create jobs but to increase R&D activity, and develop new

products, processes and services.

6 Value for money

6.1 This section of the report considers the value for money of the

SMARTCymru programme using the so-called ‘3 Es’ approach:

Economy - How much has the intervention cost; how was this decided

on; have the funds been used for the stated activities and, what

additional funds have been levered in?

Efficiency - Have activities been delivered in line with expectations

including: were the funded activities delivered in line with the plan; what

additional activities were delivered; was the cost of delivery as

expected?

Effectiveness - Have the funded activities achieved the expected

results or outcomes; what additional outcomes have been achieved, if

any; and, how has effectiveness been maximised?

Economy

6.2 The SMARTCymru programme was delivered to a total budget at the

mid-term evaluation point of £10.2m (based on eligible expenditure -

Convergence - £8.7m, and Competitiveness - £1.5m), equivalent to

£87,800 per project supported. The Welsh Government’s contribution to

the overall programme costs at the mid-term was £800,000, equivalent

to £6,900 per project supported. Public sector contributions (Welsh

Government and ERDF) represent 72 per cent of the overall programme

cost at the mid-term stage. This figure is consistent with the profile of

projects supported (with the balance of projects in the TCF and IR

phases).

6.3 The budget for the programme was determined by the SMARTCymru

team based on their experience of previous programmes, and a new

assessment of need and demand7. The Business Plans for the

Convergence and Competitiveness programmes suggest that

reasonable steps have been taken by managers to ensure the accurate

planning of the programme and its ‘economy’. This has included the

following:

7 See, for example the Business Plans for both Convergence and Competitiveness programmes.

Funding is fixed in terms of public grant aid application through

WEFO.

Positive synergies have been achieved with the Business Innovation

programme, for example through the use of the Innovation

Specialists.

Establishing intervention rates that strike an appropriate balance

between the risks of different RD&I activities, and encouraging

company investment in R&D.

6.4 As described in section 3, the implementation of the project has seen a

number of policy and organisational changes. The refocusing of the

Innovation Specialists (to sectors rather than regions), and the impasse

brought about by the proposed move to repayable finance, have not

enabled the most efficient use of staffing and resources. Although it is

difficult to quantify this at the mid-term, these factors have the potential

to impact negatively on the economy of the programme.

6.5 Despite the challenges faced in the first stage of the programme, the

recent decision to adjust the budget and targets to respond to the lack of

take-up and capacity represent sensible steps to improve economy.

Efficiency

6.6 The SMARTCymru management and delivery process involves a

relatively complex series of stages, including application support,

technical and financial appraisal, ongoing monitoring and financial

management. Despite positive feedback on the role of the Innovation

Specialist in providing advice and guidance, the main area of feedback

from companies concerned the management and delivery process. In

this respect the majority of companies felt that the application process is

overly ‘long winded’. While, for some, this was merely an inconvenience,

others highlighted the potential implications for both project and

company success and failure. Such problems were felt to be potentially

detrimental for projects in fast moving, innovative markets.

6.7 While the issues raised by the companies point towards an important

area for development, it is important to recognise the necessary ‘due

diligence’ that has to be undertaken to minimise the risk to public

funding, and the high level of skills that are needed to conduct financial

and technical appraisal of projects. These factors highlight the

importance of striking a balance between the company needs and the

need for good practice in public spending.

6.8 The challenge for the remaining period of the programme will therefore

be to minimise the impact of this important process to the companies,

through activities such as simplifying the application form and better

integrating the application and appraisal stages. This points to the need

to ensure that the application process involves, where possible, closer

integration between the appraisal (technical and financial) and

monitoring stages. It also highlights the important role of the Innovation

Specialist and the need to ensure that, where possible, relationships

with companies are maintained throughout the SMARTCymru process.

6.9 The success of the programme in generating sufficient demand is a

further area where the programme delivery efficiency could be improved.

The results of the evaluation suggest that the number of companies

entering the programme will need to be increased substantially. In this

respect many of the steps required to address this challenge are now in

place, with the move towards direct (Innovation team) promotion and

‘brand’ awareness raising, and the refocusing of the Innovation

Specialists on regional links. The evaluation results, and the scale of the

challenge, suggest that still more effective referral linkages to other

innovation and technology programmes, and organisations such as

HEIs, local authorities and so on. The Welsh Government’s sector teams

should also be an important focus, including those where sector teams

have yet to provide significant number of applications (notwithstanding

the difficulties of supporting ICT projects). For example opportunities to

generate demand from the Professional and Financial Services, Creative

and Digital should be further explored.

6.10 Post-project monitoring is a further area where management efficiency

improvements should be considered. Here, company feedback suggests

that they tend to find this element to be cumbersome. Indeed, the

Innovation team recognise that there is potential to improve efficiency by

reducing fragmentation, and to better integrate stages of the

SMARTCymru process in relation to appraisal (technical and financial),

and monitoring. To this end a dedicated monitoring unit for

SMARTCymru projects should be further considered.

6.11 A key indicator of overall programme efficiency is its return on

investment. Here the evaluation results suggest that based on total

programme expenditure8 at the mid-term point (£10.2m), the cost-benefit

ratio based on the GVA is: 1:0.44. This means that for every £1 invested

in the programme £0.44 is likely to be returned to the Welsh economy.

6.12 The GVA performance of SMARTCymru is comparable with other early

stage R&D programmes, for example, the Invest NI START (industrial

research) programme (1:0.55). GVA performance is, however, below

that identified in a final evaluation of the earlier SMARTCymru

programme (1:5.24) (Welsh Government, 2010a). This final impact data

from an earlier programme suggests that the potential GVA return will

increase further in the final stages of the current programme. It is

important to recognise, however, that the primary aim of the

SMARTCymru programme is not necessarily to create jobs but to

increase R&D activity and produce new/improved products and

processes.

Effectiveness

6.13 Against the original objectives of the SMARTCymru programme the

results of the mid-term evaluation suggest that it be considered to have

made reasonable progress, in light of the economic, policy and

organisational challenges faced, and the early stage nature of most of

the RD&I projects supported:

Creation of high quality R&D related jobs

6.14 The programme has created 31 new jobs (29 Convergence and 2

Competitiveness) by the mid-term stage. These jobs have been created

in R&D intensive companies, and are likely to be high quality in nature.

Increase businesses expenditure on R&D through the provision of financial

support to undertake innovative research and technological development

8 ERDF grant plus Welsh Government and private sector match funding.

with commercial potential, leading to new products, processes and

technologies

6.15 The programme has produced induced investment of £264,125. The

interviewed companies invested a further £282,750 on R&D on average.

If this average figure was consistent across all of the companies

supported financially, a total additional investment in R&D at the midterm

would be £27,709,500.

Encourage and support industry collaborations with other partners with

research-based organisations in carrying out industrial research and

experimental development activities

6.16 The programme has produced 14 R&D collaborations between

companies and research based organisations at the mid-term. This

relatively low number of such collaborations is largely attributed to the

presence of a range of other support mechanisms for research—industry

collaborations in Wales, for example the A4B programme, the Innovation

Voucher scheme, and the Knowledge Transfer Partnership programme.

It should also be noted that the Innovation and Technology Team has

also taken steps to address this weakness.

Measure the return on investment of supported R&D projects and provide

continuous improvement of the project management of R&D in companies

supported

6.17 Given the long term nature of the innovation process, capturing return on

investment for RD&I projects is typically undertaken some years after

the initial intervention. For this reason no ongoing measurement of

return on investment is carried out by the Welsh Government. The

results of the mid-term evaluation, however, suggest that the total

programme will contribute £1.7m in actual GVA from the mid-term

projects and £4.5m in cumulative GVA. For every £1 of Welsh

Government investment produces £5.62 net GVA.

Enable business links to other business support to optimise

commercialisation

6.18 This is an area where the synergies developed with the Business

Innovation programme and (potentially) the A4B programme provide

substantial potential to optimise commercialisation. The early role of the

Innovation Specialists, for example, in identifying companies and helping

the development of application forms is clearly important in shaping

projects. Similarly, the ongoing support of the innovation Specialists can

provide ongoing advice, and support to access further RD&I support

(Welsh Government and beyond) and funding after a SMARTCymru

project has been completed.

6.19 The results of the mid-term evaluation do, however, suggest that there is

an opportunity to strengthen synergies with the A4B project, in particular.

This will be important in addressing the programme’s aim to strengthen

collaborative links to researchers, and support collaborative R&D

activities.

Launch new or improved products, processes and services

6.20 The results of the evaluation suggest that 42 new products, processes or

services have been launched as a consequence of SMARTCymru, at

the mid-term stage. Given that studies recommend evaluating innovation

and technology projects approximately six years after the final

intervention, these figures should be treated with care. Indeed, it is

highly likely that they will grow by the end of the programme, and

beyond.

6.21 The cross-cutting themes of equality and diversity, and environmental

sustainability form an underpinning set of objectives for all European-

funded projects. In the Business Plans for Convergence and

Competitiveness targets are highlighted with respect to companies

adopting equality and diversity strategy and systems (50%), and

enterprises adopting or implementing environmental action plans (20%).

The results of the mid-term evaluation suggest that a robust process of

data collection and referral to support has been established for

SMARTCymru companies. This begins with the initial application

process, with the Innovation Specialists, and can include referral to

assistance from a range of external sources of expertise. There is

anecdotal evidence, however, that the programme demand is from a

relatively narrow equality and diversity profile. In this respect there is an

opportunity to work with the Business Innovation Programme (who fund

the Innovation Specialists) to address this weakness.

6.22 In summary, the results of the evaluation suggest that the SMARTCymru

business beneficiaries are beginning to achieve the anticipated outputs.

The early stage of many projects, however, suggests that it will be some

time before a full assessment of impact can be made. Moreover, had it

not been for the contextual challenges noted in earlier sections, outputs

would likely have been higher.

6.23 The SMARTCymru businesses generally state that they would not have

not gone ahead with the project in the same form, if it had not been for

the Welsh Government’s support. This suggests that SMARTCymru

funding is largely additional, and a good use of public funding. Demand

for SMARTCymru programme is therefore evident, however, greater

awareness and promotion is needed for final targets to be achieved.

6.24 The mid-term evaluation results further suggest that both Welsh

Government and the delivery partners have taken steps to support

delivery effectiveness. This has included a flexible approach to adapt

activities in light of changing circumstances. The cross-cutting themes

of equality and diversity, and environmental sustainability have not been

addressed in any detail.

6.25 The mid-term evaluation results point to a number of key lessons which

are important to the future success of the programme. These can be

summarised as follows:

Effective appraisal of R&D projects requires an appropriate blend of

technical and financial skills within the SMARTCymru team.

Innovation Specialists, with strong regional linkages and local

knowledge, are key to supporting companies throughout the RD&I

process, and signposting to related support.

Efficient operation of the programme relies on the ability of

managers to maintain line management control of key staff and

SMARTCymru activities, such as promotion of the programme, and

monitoring of projects and claims.

Brand awareness and effective and consistent promotion are

necessary to raise overall demand for the programme.

Strategic partners such as the universities and Finance Wales offer

important benefits with respect to raising awareness and generating

demand for the programme.

While a phased approach to SMARTCymru R&D funding

encourages companies to follow a robust and planned approach to

innovation projects, there is a need to ensure that blockages in the

pipeline of projects do not occur to the detriment of achievement of

impacts or profile of private sector funding.

Demand for the programme is likely to come from the main science

and technologically R&D intensive sectors such as advance

materials and manufacturing, and while demand is also likely to

exist in other sectors, this will be more difficult to stimulate.

In the current and recent economic climate, without Welsh

Government business R&D funding companies are likely to

underinvest in R&D.

6.26 The remaining section of report draws on these, and earlier findings, in

setting out conclusions and recommendations.

7 Conclusions and recommendations

7.1 This report provides a summary of mid-term evaluation conclusions and

recommendations for further development of the programme. The

results of this analysis suggest that the programme was established with

a sound programme logic model, based on an identified need, clear

objectives and grant activities focused on the different RD&I phases, and

anticipated targets both informed by early programme experience, and

aligned to the objectives. Clear links are also embedded in the model, to

maximise the potential for follow-on support, access to academic

expertise and so on.

7.2 At the mid-term stage the programme is some way below its anticipated

targets. In this respect it has faced a number of significant challenges

that have contributed to demand for the programme being lower than

expected. These include:

The economic downturn precipitated a R&D activity fall in Wales

(and elsewhere), indicating a lower propensity towards risk in the

business community for such activities.

The introduction of the ERP, leading to substantial organisation

reorganisation within the Welsh Government, and the loss (for a

period) of key resources such as the Innovation Specialists.

Difficulties in raising awareness and promoting the RD&I Financial

Support for Business brand, necessitating the rebranding of the

programme as ‘SMARTCymru’.

The proposed move towards repayable finance, and the difficulties

in developing the new SMARTCymru programme ‘repayable grant’

offer.

7.3 In spite of these challenges the programme has, at the mid-term stage

supported 116 projects (84 in the Convergence area, and 32 in the

Competitiveness area). These projects are typically with SMEs, with a

number of key priority sectors represented, including sectors such as the

Advanced Manufacturing and Materials sector, the ICT Sector, Life

Sciences, and Financial Services. All are consistent with the

SMARTCymru quality criteria – defined by a robust technical and

financial due diligence processes.

7.4 The performance indicators established for SMARTCymru largely reflect

those available for ERDF projects, including:

Enterprises financially supported (number)

Gross jobs created (FTE)

Investment induced (GBP)

Collaborative R&D (number)

New or improved products, processes or services launched

Products, processes or services registered

7.5 These are activity driven indicators (enterprises financially supported)

and output indicators. In this respect the indicators are focused on

capturing economic benefits (gross jobs, investment induced),

innovation benefits (investment induced, collaborative R&D), and longer

term innovative results (new products, processes or services). While

these indicators are consistent with the programme logic model set out

in section 2, they are, by their nature, quantitative. This presents

challenges for all programmes, particularly those with a focus on

behaviour change (e.g. R&D behavioural additionality), and suggests the

importance of additional evaluation research to understand these

impacts, and the potential for spillovers.

7.6 Implicit in these indicators is the long term nature of the RD&I process.

By capturing and reviewing indicators both during and at the end of a

programme, there is a danger that the indicators will underplay the true

(potential) extent of impact. This gives support to the Welsh

Government’s decision to establish an Innovation impacts team to

explore such issues through case studies.

7.7 While there is a robust process of collecting cross-cutting themes

indicator data, there is a need to address the programme’s narrow

equality and diversity demand profile. This is challenge that is shared by

its partner programme – the Business Innovation Programme, and

suggests the need for joint action if a full contribution to the cross-cutting

themes are to be met.

7.8 A related challenge is one of attributing cross-cutting theme results

specifically to SMARTCymru. This is problematic, at present, given that

the data is collected by the Business Innovation Programme. This,

again, is an area where joint discussion and action is needed.

7.9 Programme impacts at the mid-term stage are both economic- and

innovation-related. Based on findings from the business interviews (10)

these impacts have been achieved with a lower level of deadweight (30

per cent). They also suggest that Welsh Government is funding projects

that are strategically important to the companies.

7.10 Innovation impacts, at the mid-term, are limited by the early stage nature

of the RD&I projects. The business interviews do, however, point to the

majority of companies both meeting the technical objectives, and

anticipating good commercial potential for the new/improved, products,

processes and services. Companies are also investing more in R&D,

improving their capability to secure additional sources of funding from

public support programmes. The potential for innovation spillovers is

also evident, with the potential for these to produce an additional £132

thousand based on the mid-term projects.

7.11 The economic impact results suggest that the programme results to date

will contribute a net cumulative additional GVA of £4.5, and 29 net

additional jobs. This result has been achieved with comparatively low

levels of deadweight, again suggesting that project results are strongly

additional. It should be noted, however, that the main aim of the

programme has not been to create jobs but to increase R&D activity,

and develop new products, processes and services.

7.12 This performance equates to estimated return on investment, based

on cumulative net additional impacts, of:

£1 SMARTCymru total programme spending produces £0.44 GVA to the Welsh

economy

£1 SMARTCymru Welsh Government spending produces £5.62 GVA to the Welsh

Economy

SMARTCymru programme cost per project of £87,800

7.13 Return on investment (based on total programme spending) is at the

lower end of the comparators examined, however, this can largely be

explained by the comparatively short period over which the programme

activities has been evaluated.

7.14 These economic impact results have been achieved with a relatively low

level of deadweight, with the companies estimating that employment

would have been 70 per cent lower had it not been for the

SMARTCymru programme. This suggests that the projects funded by

Welsh Government are strongly additional. Of those companies that

would have gone ahead in some for the main source of alternative

finance would likely to have been bank related (loan/overdraft).

7.15 The recommendations from the mid-term evaluation are summarised

below.

7.16 Recommendation 1. Welsh Government should continue to review

longer term innovation impacts through its Innovation Impacts

programme, complementing regular programme evaluation.

7.17 Programme spend to date is consistent with the lower than expected

demand for the programme. The high proportion of projects in the early

RD&I phases is also contributing towards higher than anticipated match

funding demands for Welsh Government contributions.

7.18 Recommendation 2: Welsh Government should prioritise demand

stimulation measures, including supporting companies to move on

to the later RD&I phases in a timely manner.

7.19 ‘Brand awareness’ continues to be relatively low, and efforts are needed

to strengthen awareness of companies and other stakeholders such as

universities. Activity in this area has, understandably, been limited by the

ERP process (sector teams and so on). It is clear, however, that greater

awareness that SMARTCymru is ‘open for business’ is needed, as will

better links with potential referral bodies such as HE/FE, Business

Wales and so on.

7.20 Recommendation 3: Welsh Government should seek to ensure

businesses and stakeholders are aware that SMARTCymru is

available and open to applications.

7.21 Given the period of time lost with little or no promotional activities and

uncertainty regarding grant availability during the ERP roll out phase the

assumptions that originally underpinned the profiled and final targets are

no longer accurate. As a consequence action will be needed to review

these targets.

7.22 Discussions with the SMARTCymru team suggest that analysis has

been carried out on the pipeline of projects to determine the potential

worst case/best case/median case outcomes on project spend. The

reprofiled targets that have been developed for the programme as a

result should form the basis for a discussion with WEFO.

7.23 It will important that any discussions with WEFO take account of the

phasing of the programme, and the balance that will need to be struck

between encouraging new enterprise assists and achieving greater

innovation impacts. This balance should take account of the long term

nature of the innovation process, and the potential that projects funded

in the latter part of the programme will not achieve significant impacts in

this period.

7.24 Recommendation 4: Recommendation 4: Welsh Government

should discuss options available for SMARTCymru to revise the

final and profiled performance targets, with a focus on prioritising

projects in the ED and Exploitation phases.

7.25 There are, however, a number of areas where the management and

delivery of the programme could be strengthened to maximise the

impacts discussed above. These relate to speeding up the application

process, while maintaining appropriate due diligence procedures, for

example:

Simplifying the language used in application process

Strengthening integration across the different elements of the

SMARTCyrmu process – financial and technical appraisal, monitoring,

Innovation Specialists

7.26 Recommendation 5: Welsh Government should seek to ensure

delivery efficiency is maximised through appropriate integration

across the SMARTCymru management and delivery processes.

7.27 In relation to the RD&I phases, these are generally well aligned to the

needs of companies. Software companies appear to be one exception,

where it is difficult to meet the application criteria. Partly as a

consequence there are a number of related sectors (such as creative

industries, media and so on) where participation rates are low. Given the

importance of ICT as an enabling factor for many new innovations, there

is a need to explore whether SMARTCymru or another Welsh

Government support programme could better support such projects and

sectors.

7.28 Recommendation 6: Welsh Government should review is support

for innovation projects where ICT plays an important enabling role.

7.29 The Exploitation phase is one that very few companies have reached at

the mid-term. Of those companies that have done so, the amount of

funding available has been noted as inadequate for purposes, and a

barrier to applications. It will be important, in this respect that the Welsh

Government monitor use of this phase, to determine whether funding

can best be allocated to other programme activities / phases. Funding

for this phase, for example, could be used to develop better links to

universities.

7.30 Recommendation 7: Welsh Government should consider

refocusing funding for the Exploitation phase.

7.31 SMARTCymru helps to reduce the risks associated with downstream

public investments made by organisations such as Finance Wales.

While linkages exist between the programmes the mid-term evaluation

research suggests that there may be greater potential for exchange of

information in areas such as appraisal findings. This could help to

improve the efficiency of programme delivery for both partners.

7.32 Recommendation 8: Welsh Government should explore the

potential to strengthen synergies and knowledge exchange with

key stakeholders such as Finance Wales.

7.33 The cross-cutting themes of equality and diversity and environmental

sustainability form part of the objectives for all European-funded

projects. While robust procedures have been established for the

Innovation Specialists to collect cross-cutting themes data (at the

application stage), and to signpost to external sources of expertise, there

is a need for SMARTCymru to work with the Business Innovation

Programme more closely to both access the relevant results, and to

address shared challenges such as the potential to broaden out the

demand from underrepresented groups.

7.34 Recommendation 9: The SMARTCYmru team should work closely

with the Business Innovation Programme to share cross-cutting

themes results, and to address opportunities to improve equality

and diversity.

References

ASTUTE Wales. (n.d.). ASTUTE Wales. Retrieved October 21, 2013, from http://www.astutewales.com/en/

BIS. (2009). RDA Evaluation: Practical Guidance to Implementing the Impact Evaluation Framework.

CM International. (2009). Evaluation of the Enterprise Ireland RTI programme. CM International. (2009). Post-project evaluation of the COMPETE programme,

InvestNI. CM International. (2011). Post-project evaluation of the InvestNI Start programme. EMDA. (2010). Toolkit for the Evaluation of EMDA Strategic Programmes 2007/08-

2008/09. English Partnerships. (2008). Additionality Guide: A Standard Approach to Assessing

the Additionality of Interventions, 3rd Edition. English Partnerships. Retrieved October 21, 2013, from https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/ 191511/Additionality_Guide_0.pdf

European Commission. (n.d.). What is an SME? Retrieved October 22, 2013, from European Commission: http://ec.europa.eu/enterprise/policies/sme/facts-figures-analysis/sme-definition/

Eurostat. (n.d.). Community Innovation Survey (CIS). Retrieved October 21, 2013, from European Commission, Eurostat: http://epp.eurostat.ec.europa.eu/portal/page/portal/microdata/cis

Finance Wales. (n.d.). Finance Wales. Retrieved October 21, 2013, from http://www.financewales.co.uk/

Frontline Consultants. (n.d.). Evaluation of the IDA Ireland R&D Grant scheme. Frontline Consultants. (n.d.). R&D Grant Final Evaluation for Scottish Enterprise. HM Treasury. (2013). Supplementary Green Book Guidance: Optimism Bias.

Retrieved November 18, 2013, from https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/ 191507/Optimism_bias.pdf

Office for National Statistics. (2013, November). Business Enterprise Research and Development, Research and Development in UK Businesses, 2012. Retrieved October 21, 2013, from http://www.ons.gov.uk/ons/dcp171778_337993.pdf

Office for National Statistics. (2013, December). Statistical bulletin: Regional Gross Value Added (Income Approach), Regional GVA NUTS1 dataset. Retrieved October 21, 2013, from Office for National Statistics: http://www.ons.gov.uk/ons/dcp171778_345191.pdf

PACEC. (2011). Evaluation of the TSB Collaborative R&D programmes. Retrieved October 22, 2013, from https://www.innovateuk.org/documents/1524978/1814792/Evaluation+of+the +Collaborative+Research+and+Development+Programmes+Final+Report/e09 dbba0-1cfb-4607-973f-ae9caa73f66d

PricewaterhouseCoopers. (2009). Impact of RDA Spending: National Report Volume 1. Department for Business, Enterprise and Regulatory Reform. Retrieved October 21, 2013, from http://webarchive.nationalarchives.gov.uk/+/http://www.berr.gov.uk/whatwed o/regional/regional-dev-agencies/Regional%20Development%20Agency%20Impact%20Evaluation/pa ge50725.html

Statistics Wales. (2014, February). Statistics Wales. Retrieved October 22, 2013, from https://statswales.wales.gov.uk/Catalogue/Business-Economy-and-Labour-Market/People-and-Work/Employment/Jobs/Whole-Workforce/WorkplaceEmployment-by-Area-Year

Welsh Assembly Government. (2008). European Structural Funds 2007-2013 Strategic Framework: Innovation, R&D and Technology, ERDF Convergence: P1T1 ERDF Competitiveness. Retrieved October 21, 2013

Welsh European Funding Office. (2010). West Wales and the Valleys Convergence Programme European Regional Development Fund 2007-2013. Retrieved October 21, 2013, from http://wales.gov.uk/docs/wefo/publications/convergence/erdfoperational/1106 01erdfconvopen.pdf

Welsh European Funding Office. (2012a). Revised Business Plan Outline: RD&I Financial Support for Business, Convergence.

Welsh European Funding Office. (2012b). Revised Business Plan, RD&I Financial Support for Business, Competitiveness.

Welsh European Funding Office. (n.d. b). Progress of 2007-2013 programmes: RD and I Finance for Business Competitiveness. Retrieved October 21, 2013, from Welsh European Funding Office: http://wefo.wales.gov.uk/programmes/progress/searchprojects/80628?lang=en

Welsh European Funding Office. (No date a). Progress of 2007-2013 Programmes -RD and I Finance for Business, Convergence. Retrieved October 21, 2013, from Welsh European Funding Office: http://wefo.wales.gov.uk/programmes/progress/searchprojects/80504?lang=en

Welsh Government. (2010). Economic Renewal: A New Direction. Retrieved October 21, 2013, from Welsh Government: http://wales.gov.uk/topics/businessandeconomy/publications/economicrenewa l/?lang=en

Welsh Government. (2010a). Impact Evaluation Report. Internal Report. Welsh Government. (2012, September). Academic Expertise for Business Project

Directory: Directory of projects supported through the Welsh Government's European funded Academic Expertise for Business (A4B) programme, 2008-2012. Retrieved October 21, 2013, from Expertise Wales: https://www.expertisewales.com/resource/A4B%20Directory%20(English).pd f

Welsh Government. (2012). Business Innovation. Retrieved October 21, 2013, from Business Wales, Welsh Government Support for Innovation: https://business.wales.gov.uk/sites/default/files/documents/Growing%20a%20 business/Business%20Innovation%20Brochure_Eng_Nov%202012_growing. pdf

Welsh Government. (2013). SMARTCymru: Research, Development and Innovation Funding. Retrieved October 21, 2013, from https://business.wales.gov.uk/sites/default/files/documents/Growing%20a%20 business/SMARTCymru%20Brochure%20Eng_growing.pdf

Welsh Government. (n.d.). SMARTCymru Monitoring data. Welsh Government. (n.d.). Specification for Mid-term Evaluation of SMARTCymru

RD&I Fianancial Support for Business Programme. Research and Evaluation Framework Agreement No. C255/2008/09.

Welsh Government. (n.d.). SMARTCymru monitoring data, Convergence & Competitiveness. Retrieved September 2013

Welsh Government. (n.d.). The Di Minimus Regulation. Retrieved October 21, 2013, from http://wales.gov.uk/topics/businessandeconomy/stateaid/sarules/deminimis/?la ng=en

Annex

A. Detailed methodology

7.35 The mid-term evaluation of SMARTCymru incorporated a range of

methodological stages. This included a number of work stages designed

to review the management and delivery process, impacts achieved at

the mid-term, and their value for money. The main focus was to make

use of the available documentary and monitoring information collected

by the programme, and to minimise the need for new fieldwork. This

approach formed part of the Welsh Government’s requirement for the

mid-term evaluation.

7.36 The mid-term evaluation began with a series of stakeholder interviews

including Welsh Government SMARTCyrmu programme managers and

other relevant staff (see Annex II for details of interviewees). These

interviews were designed to identify the key features of the programme

logic model, and to understand expectations for the mid-term evaluation.

A subsequent half-day workshop was facilitated with the main delivery

staff for the programme (the Innovation Specialists, finance and

monitoring staff).

7.37 The stakeholder interviews and management and delivery workshop

were complemented by a documentary review. This comprised analysis

of programme documentation, including the SMARTCymru Business

Plans (Convergence and Competitiveness), appraisal and monitoring

forms and so on. Alongside this analysis comparator impact data was

collected to contextualise the economic and innovation impact results.

7.38 The focus of the company beneficiary analysis was to analyse the

monitoring data collected by the Welsh Government as reported to

WEFO. This included both economic and innovation performance

indicators, as well as programme spend/income data, including (Welsh

Government, n.d.):

Enterprises financially supported (number)

Gross jobs created (FTE)

Investment induced (GBP)

Collaborative R&D (number)

New or improved products, processes or services launched

Products, processes or services registered

7.39 This monitoring data was used in developing an understanding of the

mid-term profile of participation in SMARTCymru, as well as

achievements against the forecasted targets.

7.40 A series of ten telephone interviews were conducted with companies

that had completed one or more of the SMARTCymru RD&I phases.

These interviews were not intended to provide a fully representative

sample but to highlight the diversity of participation and experiences in

SMARTCymru, and, to ‘signpost’ attention towards some of the impact

areas emerging and key issues for the remaining part of the programme.

The interviews were also used to inform the assumptions used in the

economic impact analysis (see below for more detail).

Economic impact calculations

7.41 The economic impact assessment and calculations draws on the

methodology employed in the ‘Impact of RDA Spending’

(PricewaterhouseCoopers, 2009), BIS Practical Guide (BIS, 2009) and

the English Partnerships Additionality Guide (English Partnerships,

2008). This methodology calculates economic impact from the (WEFO)

reported jobs created indicator, and uses a conversion rate for

determining GVA impact. This produces three levels of net GVA impact

Actual benefit – defined as the net additional GVA produced by the

mid-term point

Cumulative benefit – defined as the actual net additional GVA plus an

estimate of the persistence of these impacts.

Future benefit – the estimated future annual net additional GVA if the

final targets are achieved

7.42 The data that is produced by this method should, given the mid-term

nature of the results, be considered as an estimate, and treated with

appropriate caution.

7.43 The following steps were followed in calculating the Actual achieved

GVA impact from SMARTCymru at the mid-term:

Calculate gross jobs created – the number of FTE gross jobs created

is taken from the SMARTCymru (Convergence and Competitiveness)

indicator data (to June 2013). To estimate the build-up of these jobs it

is assumed that 7.75 FTE jobs were created in 2011, growing to 15.5

FTE jobs in 2012, and 31 FTE in 2013.

Convert gross jobs to net jobs for all three years using the

additionality factors identified in the business interviews (described in

section 5).

Summary results of the net additional jobs calculations are set out in

table 23 below.

Table 23. Net additional jobs (Actual)

Estimated Gross Jobs in Year 1 (2011)

Estimated Gross Jobs in Year 2 (2012)

Gross Jobs in Year 3 (2013)

Gross Impacts 7.75 15.50 31.00

less deadweight (30%) 21.70

less displacement (0%) 21.70

less substitution (0%) 21.70

less leakage (10%) 19.53

Total Net Additional Impacts

plus multiplier (150%) 29.30

7.44 The following steps were followed in calculating the Cumulative

achieved GVA impact from SMARTCymru at the mid-term:

Calculate GVA per job for Wales – Using the Wales Workplace

based GVA at current basis prices for 2012 (Office for National

Statistics), divide by employee numbers in Wales for 2012 from the

Statistics Wales data (Statistics Wales, 2014) to calculate the GVA per

job in Wales. This figure works out as £35,886 per job in 2012.

Similarly, GVA per job for 2011 is calculated by taking Workplace GVA

in 2011 and diving by the employee numbers in Wales for 2011

(£35,112).

The GVA for Wales in 2013 is currently unavailable. Therefore, to

calculate the GVA per job in 2013, it is assumed that the 2012 figure

for Workplace GVA for Wales remains constant, but is discounted by

3.5% to take account of inflation. This figure is then divided by the 2013

employee numbers in Wales to get GVA per job in 2013.

Calculate the gross GVA achieved by the programme – multiply the

number of gross jobs reported each year by the GVA per job in year for

Wales.

Convert Gross GVA to net GVA – Using additionality factors derived

from the business interviews calculate the net additional GVA.

Summary results of the Actual net additional GVA calculations are set

out in table 24 below.

Table 24. Net additional GVA – Actual

Wales GVA and Workforce Jobs Data (000s)

GVA per job (£000s)

2011 GVA generated from jobs (£000s)

2012 GVA generate d from jobs (£000s)

2013 GVA generat ed from jobs (£000s)

Total Actual GVA generatedfrom jobs (£000s)

Wales Workplacebased GVA at current basic prices 2011 (£,000)

46,450,000.00 35.11 272.12

Wales Workplace based GVA at current basic prices 2012 (£,000)

47,344,000.00 35.89 556.23

Wales Workplace based GVA (Present Value) 2013 (£,000)

45,742,995.17 32.98 1,022.37 1,850.72

Gross GVA Impacts

272.12 556.23 1,022.37 1850.72

less deadweight (30%)

190.48 389.36 715.66 1295.51

less 190.48 389.36 715.66 1295.51

displacement (0%) less substitution (0%)

190.48 389.36 715.66 1295.51

less leakage (10%)

171.44 350.42 644.10 1165.96

plus multiplier (150%)

Net Additional

257.15 525.64 966.14 1748.93

7.45 In order to understand the cumulative and future potential benefits at the

mid-term the following assumptions are made for the persistence of the

benefits: how many years the stream of benefits is expected to persist.

Cumulative impacts- is calculated based on an estimated persistence

of the mid-term (actual) impacts – three years. This is consistent with

the BIS (BIS, 2009) guidance on innovation persistence. To calculate

persistence the 2013 net additional GVA is multiplied by three, giving a

total persistence of £2.9m. This is then added to the Actual net

additional GVA to give a total project net additional GVA of £4.6m.

Present Value of GVA impacts – the net additional GVA impact

(£4.6m) is then adjusted to produce the net cumulative additional GVA

of £4.5m with the base year 2013 (i.e. net present value) (EMDA,

2010). Summary results for the net cumulative additional GVA

calculations are set out in table 25 below.

Table 25. Net additional GVA – Cumulative

Net Additional GVA

Actual Impacts Persistence of Impacts

Discount Rate Net Present Value of Persistence

Year 1 2011 £ 257.15 1.07 £ 275.47

Year 2 2012 £ 525.64 1.04 £ 544.03

Year 3 2013 £ 966.14 1.00 £ 966.14

2014 £

966.14 0.97 £ 933.47

2015 £

966.14 0.93 £ 901.91

2016 £

966.14 0.90 £ 871.41

Total Persistence of Impacts

£ 2898.43

£ 2,706.78

Total Net Cumulative Additional GVA (£000s)

£ 4,647.36

£ 4,492.43

7.46 The following steps were followed in calculating the Future potential

GVA impact from SMARTCymru:

Calculate future potential gross GVA – by taking the number of FTE

gross jobs projected by the end of the project and multiplying by the

GVA per job in 20139.

Calculate the Net additional future potential GVA- Using the

additionality weighted factors explained above calculate potential net

additional GVA. The potential net additional GVA is then adjusted to

produce the future estimate of net additional annual GVA of £3.4m with

the base year 2013 (i.e. net present value). Summary results of future

annual potential net GVA is set out in table 26 below.

Table 26. Future potential impact per annum

Future Potential Future Estimate (£000s)

Wales GVA 2013 £ 45,742,995

Wales GVA Per Job £ 32.98

Future Estimate Gross GVA Impactfrom SMARTCymru

£ 3,858.64

Future Estimate Net GVA Impact from SMARTCymru in 2015

£ 3,646.41 Present Value of Future Net GVA Impact from SMARTCymru (in 2013 prices) £ 3,403.97

9 For the purpose of this calculation the final target for jobs created has been reduced to 117 (72 Convergence jobs, and 45 Competitiveness jobs) from 195 to

account for revised targets developed by the programme team. It should be noted, however, that these revised targets had not been agreed with WEFO at the

time of the mid-term evaluation, and have therefore not been included in section 5.

B. Mid-term business interviewees

7.47 Ten business were interviewed as part of the mid-term evaluation. This

included:

NextGen Venturi

Assay Innovation

Clyne Energy

MC Diagnostics

Cyden

The R&D Factory

Eysys

Norland Technology

Structured Productivity Solutions

Dulas