Beyond surveys The research

frontier moves to the use of

administrative data to evaluate

RampD grants

Michele Morris1 and Oliver J Herrmann2

1Ministry of Environment Evaluation team Wellington 6143 New Zealand and 2Ministry of

Business Innovation amp Employment Evaluation team Wellington 6011 New Zealand

Corresponding author Email oliverjherrmanngmxde

Evaluation has relied on survey data sources Different fields of social sciences (political sciencesociology and economics) were revolutionized using modern survey methods and techniques foranalysing these data However the research frontier moves to the use of administrative dataAdministrative data are highly preferable to survey data because (1) full population files aregenerally available and offer much larger sample sizes (2) administrative data have a longitudinalstructure that enables evaluators to follow individuals over time and address many critical policyquestions (3) administrative data do not suffer from high rates of non-response attrition andunder-reporting This article presents results from an evaluation of publicly funded RampD assistanceprovided to New Zealand firms The grants are intended to lead to additional RampD undertaken bythe firm which in turn is expected to increase the economic performance of a firm We use theStatistics New Zealand prototype Longitudinal Business Database (LBD) which contains highquality and comprehensive administrative firm-level data The problem of evaluation is thatwhile the programmesrsquo impact can truly be assessed only by comparing actual and counterfactualoutcomes the counterfactual is not observed So the challenge of an evaluation is to create aconvincing and reasonable comparison group for assisted firms in light of this lsquomissing datarsquo Ourmethodology involves matching firms that received assistance to comparable unassisted firmsbased on firm characteristics We then compare changes in performances of the assisted

group with the group of matched unassisted firms

Keywords RampD administrative data firm-level data grants matching control group smalland medium enterprises (SME)

1 Introduction

This article focuses on the quantitative impact of govern-ment RampD assistance to firms The problem of evaluationis that while the programmesrsquo impact (independent ofother factors) can truly be assessed only by comparingactual and counterfactual outcomes the counterfactualcannot be observed In light of this lsquomissing datarsquo oneapproach to measuring program outcomes is to create aconvincing and reasonable comparison group for assistedfirms Our methodology involves matching firms thatreceived assistance to comparable unassisted firms basedon firm characteristics We then compare changes in

performances of the assisted group with the group ofmatched unassisted firms

While this approach has been used in other countries ithas only recently been applied in an evaluation in NewZealand Qualitative evaluation methods that elicit infor-mation from programme participants often attribute all ormost changes in firm performance to the government pro-gramme While important in describing how RampD addsvalue to the business this methodology does not take intoaccount the fact that many firms that seek governmentassistance are already growing faster and performingbetter than an average firm To overcome this problemassisted firms were matched with a comparable group of

Research Evaluation 22 (2013) pp 298ndash306 doi101093resevalrvt020Advance Access published on 17 October 2013

The Author 2013 Published by Oxford University Press All rights reserved For Permissions please email journalspermissionsoupcom

unassisted firms from the Statistics New Zealand proto-type Longitudinal Business Database (LBD)1 whichcontains high quality and comprehensive firm-level datafrom 2000 to 2008

The introduction also describes the general evaluationframework the available data and the method adopted toremove selection bias In Section 2 we present an overviewof RampD assistance in New Zealand and describe the modelspecification and our findings Results are presented inSection 3

11 The evaluation framework

In this article we assess the direct impact of receivingfinancial government assistance on the performance offirms At the heart of evaluation is the following Inseeking to know the impact of a programme on a firmwe wish to compare what happens if the firm receives gov-ernment assistance (in the language of the literaturereceive the treatment) to what would happen otherwiseIf we call the first Y1 and the second Y0 then the treatmenteffect for each firm i at any time t is defined as the differ-ence between its potential outcomes

it frac14 Y1it Y0

it eth1THORN

Depending on the time the outcomes of interest mightbe intermediate outcomes such as increases in a firmrsquosRampD activity or final outcomes such as improvements inproductivity or revenues The fundamental evaluationproblem arises because we cannot observe both whatwould happen if the firm received assistance (Y1) andwhat would happen if it did not (Y0) The outcome thatwe do not observe is called the lsquocounterfactualrsquo The meas-urement of impact compared with what would havehappened otherwise is sometimes called lsquoadditionalityrsquoEvaluation of government assistance should try to testfor attributable impacts including the additionality ofthese impacts over what would otherwise have occurredwithout the assistance (Bartle and Morris 2010)

The core evaluation issue is set out in Fig 1 Consider afirm that produces a at a prior to assistance (at time t1)After receiving assistance (at time t2) the firmrsquos sales riseuntil it produces at point d One simple way to measure theimpact of the assistance would be to compare the firmbefore (a) and after (d) assistance This is the same asassuming that the firm would not grow (ie it wouldhave been at b at time t2) If we did this we would inferthat the impact was measured by the distance bd That is itis the assistance increased sales by Y1

YrsquoHowever this ignores what was happening to the firm

already (as well as the other changes that affected the firmsince assistance) We can see from the figure that thefirm was already on an upward sales path If the firmhad maintained its pre-assistance trajectory it wouldhave ended up at d anyway The difference between azero impact and db in our figure is significant

If somehow we knew that the firm would have actuallyended up at c if it had not received assistance we couldclearly identify the impact as the distance dc At time t2the additional sales the firm enjoyed as a result of receivingassistance is Y1

Y0 The total additional sales enjoyed bythe firm is the shaded area between the two lines describingwhat happened after the firm received assistance and whatwould have happened if it did not receive assistance

During an evaluation the counterfactual is not observedbut must be estimated One way to do this is to find asuitable comparison group of firms and compare theoutcomes of the firms receiving assistance with those ofthe control group However a group of firms receivingassistance should not be compared with another randomgroup selected from the business population Firms thatreceive government assistance could be higher performingthan the average firm even before receiving assistance (seealso MED 2009 de Beer et al 2010) Therefore a simplecomparison of outcomes between assisted and non-assistedfirms would reveal a spuriously high treatment effect forreceiving assistance because it would also include pre-existing differences in firmsrsquo outcomes

Shaded area = impact

Firm with

assistance

Firm without

assistance

a b

c

d

Outcome Y (eg sales)

Time t1 t2

Zero growth

O

Y0

Y1

Y

Figure 1 Defining a counterfactual

Beyond surveys 299

The primary methodological issue is to overcome thisselection bias (see also World Bank 2010) Fortunatelywe have a rich data set to allow for a wide range of char-acteristics to be incorporated in the participation model

12 Economic micro-data in New Zealand

The main unit of analysis in micro-data is the firm or a jobseeker in the case of labour market analysis For somepurposes micro-data have benefits over survey databecause (1) full population files are generally availableand offer much larger sample sizes (2) administrativemicro-data have a longitudinal structure that enablesevaluators to follow individual firms over time and (3)administrative micro-data do not suffer from non-response attrition and under-reporting Access to micro-data is critical for cutting-edge empirical research and forpublic policy evaluation

The data for the analysis in this report come fromStatistics New Zealandrsquos prototype LBD The LBDcontains business-related data for financial years 2000ndash2008 from a number of sources The main unit ofanalysis is the lsquoenterprise levelrsquo where the enterprise isdefined as a business or service entity operating in NewZealand The spine of the LBD is the LongitudinalBusiness Frame (LBF) which records firm characteristicsand changes in these characteristics over time The LBF isable to identify the predominant industry affiliation of afirm [Australian and New Zealand Standard IndustrialClassification (ANZSIC) code] foreign ownership inter-ests and firmsrsquo connections through some kind of report-ing arrangement

Several administrative data sources are also attached tothe LBF These include Goods and Services Tax (GST) taxreturns (IR4) financial accounts (IR10) and aggregatedPay-As-You-Earn (PAYE) returns provided by the InlandRevenue Department and Customs merchandize tradedata Several Statistics New Zealand business surveys arealso attached including the Annual Enterprise Surveywhich is the primary data source for calculating theNational Accounts the Business Operations Survey andthe Research and Development Survey All data in theLBD are annualized to firmsrsquo actual balance date andthen assigned to the closest year ending 31 March Thedatabase continues to evolve over time It is described inmore detail in Fabling (2009) and de Beer et al (2010)

The LBD also includes participation data for businessassistance schemes administered by New Zealand Trade ampEnterprise (NZTE) and Foundation for Research Scienceand Technology (FRST) Te Puni Kokiri (TPK MaoriDevelopment) and New Zealand Venture InvestmentFund NZTE FRST and TPK provided a list of firmsthat had received assistance with information on theduration and amount of assistance These details werematched with GST numbers (where supplied) or probabil-istically linked (on name and contact details) to the LBF

These records were matched where possible to firm recordsin the LBD with high matching rates

2 Government RampD assistance An example

Innovation is the main determinant of productivity (andhence economic) growth in developed economies RampDexpenditure is a major determinant of this2 The socialrate of return for RampD tends to be higher than theprivate rate Hall et al (2010) Hall and Ziedonis (2001)This creates insufficient incentives for firms to invest inRampD from the perspective of society as a whole Becauseof this governments intervene to stimulate RampD spendingIt is difficult to calculate the impact of government RampDassistance because firms with a high likelihood of benefit-ing from and undertaking RampD self-select into treatmentBecause of this simple studies tend to overstate thebenefits of assistance It is therefore important to use ap-propriate techniques such as identifying a control group ofsimilar firms One distinguishing feature of this article isthe focus on how the impact of assistance varies acrossfirms Economic theory and evidence suggest that smalland larger firms differ in the amount and nature of theirinnovation behaviour including RampD To design effectivepolicy therefore it is important to understand how theimpact of assistance varies across firms

In this section we describe the New Zealand schemesthat are the subject of this evaluation It is important tounderstand the types of firms that receive assistance andhow they compare with the general population of firms

The RampD programmes included in this evaluation areknown as Technology New Zealand (Tech NZ) Some ofthese programmes are no longer current Also the detailedobjectives and implementation of most of these pro-grammes have evolved over time Tech NZ represents thebulk of publicly funded RampD assistance targeted directlyto firms in New Zealand New Zealand firms also receiveindirect forms of government assistance throughuniversities and Crown Research Institutes undertakingpublicly funded RampD to assist long-term industry needsThis type of indirect government assistance by public pro-viders is not evaluated in this report We also do notevaluate the services provided to firms by Tech NZ staffincluding the Global Expert scheme which helpsbusinesses link to local and global RampD expertise

The Tech NZ programmes are split into two categoriesassistance to build RampD capability and assistance forRampD projects for firms with more highly developedRampD capability

Annual Funding for Tech NZ has grown from an initialvalue of $112 million in 1996ndash7 financial year to $509million in 2008ndash9 For methodological and data availabil-ity reasons this evaluation focuses on firms first receivingTech NZ assistance between 2002 and 2008 (inclusive)Table 1 displays summary information

300 M Morris and O J Herrmann

The expected long-term outcomes for both types of as-sistance (Capability Building and Project Funding) reflectthe objectives of Tech NZ enhancement in business per-formance at the firm industry and economy level throughincreased private sector RampD However it is reasonable toexpect that there will be differences in the short-termoutputs and intermediate outcomes of the two differenttypes of assistance Certainly both are expected to buildRampD capability in the short-term However projectfunding in particular is also expected to result in moretangible outputs such as new or improved productsprocesses andor services

Both categories of assistance schemes target a widerange of firms at different stages of their life cyclemdashfromearly stage to maturity The distinguishing characteristicbetween the two categories is the type of activity fundednot the type of firm There is an additional emphasis ontargeting firms with potential for high growth with ProjectFunding This means that a mature firm may engage incapability building in a new area with a goal of increasingtheir understanding and technological awareness A start-up company may apply for and receive significant projectfunding if it can demonstrate (among other things) that theidea and has potential to general significant future revenuefor the firm

21 The evaluation sample

Table 2 shows the number of firms that first receivedRampD assistance between 2002 and 2008 distributed bytype of assistance and year of entry into a governmentRampD programme The number of new entrants in allcategories trends down between 2002 and 2008 This isassociated with an increase in the mean grant size (notshown) as the total available funding increased over thisperiod

The population is drawn from lsquoprivate for profitrsquo firmsand excludes households ANZSIC division M(Government Administration and Defence) and firmslocated offshore Only firms that are considered to beeconomically active are included firms not assigned an

ANZSIC industry classification are dropped We only

include treated and untreated firms for which we have a

full set of data to calculate matching variables in the

pretreatment year Because our matching model includes

growth variables we require data 2 years before a firm first

receiving assistance The LBD starts in 2000 so we can

only examine the impact on firms that first receive RampD

assistance from 2002 onwards A large number of treated

firms has been dropped primarily because they had zero

total employment in the 2 years preceding their first year of

government RampD assistancePrevious research results showed that firms that receive

government assistance are on average higher performing

than the average New Zealand firm even before receiving

assistance (MED 2009 de Beer et al 2010) Firms that

seek and receive Tech NZ assistance have higher sales

employment capital-labour ratios and labour prod-

uctivity (LP) compared with the rest of the population

of firms that did not receive any government RampD assist-

ance Assisted firms appear to have slightly lower

multi-factor productivity (MFP) than unassisted firms

Interestingly there does not appear to a large degree of

differentiation between firms that receive Capability

Building and Project Funding assistance at least in

the year before first participating in a Tech NZ

programmeThere are other ways in which assisted firmsrsquo perform-

ance may differ from the average New Zealand firm For

example if only firms with potential for high growth

receive Project Funding assistance then the evaluation

will tend to overstate the impact of the assistance if a com-

parison is made with the remainder of the business popu-

lation This is an example of selection bias in the impact

estimate The appropriate comparison group would be

firms that also have the potential for high growth but

have not received assistance

Table 2 Number of firms receiving RampD assistance by year of entry

and type of assistance

Year of first RampD assistance Firms receive RampD assistance only

Capability

Building

Project

Funding

Both types

of assistance

2002 309 60 57

2003 249 108 45

2004 255 63 36

2005 186 57 21

2006 147 39 18

2007 144 54 18

2008 126 45 9

Total number of firms 1413 426 207

Share of total number of firms 69 21 10

Share of total dollars dispersed 12 63 25

Table 1 Overview of Tech NZ 2002ndash8

Tech NZ programmes Total grant

payments

(millions $)

Number

of grantsa

Technology for Business Growth (TBG) 1560 945

Grants for Private Sector Research and

Development (GPSRD)

572 1086

Technology Fellowships (TIF) 388 747

TECHLINK 78 2850

aNumbers in this and following tables have been randomly rounded to base 3 to

protect confidentiality

Beyond surveys 301

22 Model specifications and results

Our analysis uses a propensity score matching approach

see also Rosenbaum and Rubin (1983) HeckmanIchimura and Todd (1998) Heckman et al (1998)Caliendo and Kopeinig (2008) We predict the probabilityof receiving RampD assistance using a probit regressionmodel based on characteristics before assistance Ourchoice of matching variables is based on the Tech NZ pro-gramme selection criteria and evidence from the literatureespecially Fabling (2008) Fabling and Grimes (2009)

Fabling and Sanderson (2010)Matching is based on predicted probability from a

probit regression of the treatment variable on characteris-tics in the previous year The coefficients on employmentcapital intensity exporting and RampD activity are signifi-cant and have the expected signs Larger firms and firmswith higher capital intensities have a higher likelihood ofreceiving RampD assistance There is also a significant non-linear dependence on firm size so that the probability ofreceiving RampD assistance does not continue to increase

with firm size Firms that export goods are also morelikely to receive assistancemdashthe coefficients are similarwhether the firm is in a goods exporting industry or notRampD activity in the 2 years before first time assistance isalso a strong predictor of receiving RampD assistance Theseresults confirm our expectations that higher performingfirms self-select (or are preferentially selected by fundingagencies) to request RampD assistance

There is no evidence of firms gearing up beforeundertaking a government RampD project Growth ratevariables are not statistically significant except in the

case of firms seeking Capability Building assistancewhere lower growth rates in MFP increase the likelihoodof receiving assistance all other things equal Youngerfirms are also more likely to receive Capability Buildingassistance Foreign ownership and group status onlyimpacts on the probability of receiving Project Fundingforeign owned firms are less likely and firms in groupsmore likely to receive Project Funding

The matching models show that the performance ofassisted firms is higher than the average New Zealand

firm even before the assistance begins The questionremains whether the RampD assistance results in any add-itional impact on firm performance We obtain causal es-timates of the effect of RampD assistance on firmperformance by using the estimated propensity scoresfrom the matching model to match assisted firms to com-parable control firms and then compare the changes infirm performance between treated and matched controlfirms

The rest of this section presents the results for our threedifferent model specifications With our first model we

investigate the individual impacts of Capability Buildingand Project Funding assistance These programmes eachhave a different focus Capability Building focuses on

growing firmrsquos RampD capability Project Funding focuseson assisting firms that already have that capability toundertake RampD projects Because of this the two sets ofprogrammes would be expected to have different impacts(particularly on intermediate outcomes and the mechan-isms whereby ultimate outcomes would be influenced)

In our first model we divide firms into three groupsThose that received Capability Funding those thatreceived Project Funding and those that received no gov-ernment assistance at all (Fig 2) Because the focus is onidentifying each programmersquos individual impact weexclude firms that received both from the sample in thismodel

We found that Capability Building assistance has apositive and sustained impact on employment RollingMean Employment (RME) The coefficients are 5meaning that on average firms that received CapabilityBuilding assistance subsequently grew their employmentto be 5 higher than similar matched firms over thesame period The impact was evident up to 3 years followinga firm first receiving Capability Building assistance Most ofthe differential employment growth happens at the start ofthe assistance with only a small increase after 3 years (from47 to 67) The lack of significant impact on RME after 4years may partially reflect the decline in sample size and thecorresponding increase in standard errors (Fig 2)However the lack of significant impact after 2 years is dif-ficult to explain There is a decline in sample size between 1and 2 years however the impact is significant after 3 yearsand the sample is even smaller then This pattern may belinked to the intermittent nature of RampD activity particu-larly for firms that are just starting build RampD capacity

Analysis shows an immediate impact on sales in the firstyear after first receiving Capability Building assistance butno impact for later years (although the coefficients arepositive they are not statistically significant) There is noimpact on LP although the coefficient at lags of 4 years isgetting close to becoming significant If the differentialgrowth in value added between treated and matchedcontrol firms remained constant then the significant dif-ference in employment growth might lead to negativevalues of LP In fact LP coefficients are positivealthough not significant implying that value added musthave also increased although not so much that we see sig-nificant impact on LP There is no significant impact onMFP for the first 3 years following the first year of

Figure 2 Model 1mdashCapability Funding and Project Funding

302 M Morris and O J Herrmann

assistance However we do see a significant positiveimpact on one of our final outcomes MFP at 4 yearsfollowing first receiving Capability Building assistance

There is a large reduction in the number of observationsfrom zero lag to lags of 4 years Most of this reduction indata is a reflection of our relatively short time series re-sulting in a smaller number of possible observations at lagsof 4 years compared with no lag However some of thisreduction will be due to firm attrition It is possible thatour treatment effect estimates are affected by a change incomposition of treated firms over time eg if high-per-forming firms are more likely to survive then our estimatesat longer time lags may be positively biased owing to at-trition of weaker firms To test this we repeated theanalysis using a common sample that only included firmsthat were present in all years in order to untangle changesin the composition of the assisted firms over time fromwithin-firm changes in performance The commonsample results showed the same time dependence inimpact on MFP as the full sample there is no statisticallysignificant impact until 4 years after first receiving assist-ance This suggests that attrition bias is not the cause ofthe increase in significance for MFP at later years Theevidence of a significant long-term impact on MFP fromCapability Building is encouraging because higher firmproductivity is a final rather than an intermediateoutcome MFP could be improved by firms adoptingbetter business practices or utilizing resources in more ef-ficient ways as a result of receiving RampD assistance If thisis what is happening then we would expect to see evenhigher returns in the future

For Project Funding assistance there are no causaleffects on any of firm performance measures at any timelag Most of the employment coefficients are positive butnot statistically significant All the productivity coefficientsare negative and not significant The results seemcounterintuitivemdashProject Funding involves large sums ofmoney and is directed to firms that are embarking on sig-nificant RampD projects that have been found to producetangible results (eg Infometrics 2009) Yet surprisinglyanalysis shows no impact on any firm performancemeasures for up to 4 years following project approvalThis does not mean that the RampD projects undertakenas a result of receiving assistance have been unsuccessfulBased on previous evaluation evidence of Tech NZ it ishighly likely that the projects improved the performance offirms following receipt of the grant However our resultssuggest that a comparable group of firms would have hadsimilar improvements in firm performance (possibly as aresult of undertaking privately funded RampD) regardless ofwhether they received Project Funding assistance

Why do we see significant impacts for CapabilityBuilding assistance but not Project Funding Is it relatedto the types of firms that receive Capability Building egbecause firms are less likely to have undertaken RampDbefore receiving Capability Building We examine this

question in our next two specifications We also examinethe possibility that the lack of significance is associatedwith the smaller sample size for Project Funding (30of the Capability Building sample size)

For our next two specifications we pool both types ofassistance to increase our sample size and examine the in-fluence of firm size and prior RampD activity on our esti-mates (We can now include the firms that received bothCapability Building and Project Funding assistance whichwe previously excluded to get a clear picture of theseparate impacts) To do this we need to assume thatthe probability of receiving assistance and the time-de-pendent impact on firm performance does not depend onthe type of RampD assistance We see some justification forthis assumption in the similarity of firm characteristicsbefore receiving either Capability Building or ProjectFunding assistance and in the similarity of the matchingmodels

We might also expect the impact of assistance may varyis by firm size Larger firms may be at different stages oftheir development and also gain from economies of scaleAn important aspect of economies of scale is the ability tospecialize and to conduct potentially resource-intensiveactivities such as RampD There is also literature thatsuggests that larger firms are more likely to conduct par-ticular types of RampD Coombs and Georghiou (2002)

In this model we divide both the group of assisted firmsand unassisted firms by size in terms of employment(Fig 3) Our split point is chosen so as to split the groupof assisted firms in half Because this reduces the samplesize of our assisted firms we pool the Capability Buildingand Project Funding recipients into one group This meanswe can also include firms that receive both CapabilityBuilding and Project Funding This means that in ouranalysis we do two comparisons In the first wecompare small firms receiving Capability Building andorProject Funding assistance with small firms receiving nogovernment assistance In the second we compare largefirms receiving either type of assistance with large firmsreceiving no government assistance

We split the sample of all assisted firms into two equalgroups based on total employment (RME) The splitoccurs at RME=623 There is a clear pattern from ouranalysis we see some positive and significant impacts on

Figure 3 Model 2mdashVariations in impact by firm size

Beyond surveys 303

firm performance for the small firms but no impact for

large firms In fact we see significant negative impacts on

productivity at two time lags for the large firm group For

firms in the small size group we see differential growth in

employment LP and MFP after 4 years between assisted

and unassisted firms The estimates are not significant at

shorter time lags except for employment but coefficients

are positive from lags of 1ndash3 years for employment and

LP In contrast all the productivity coefficients are

negative for the large firm group although employment

coefficients are positive The lack of significance in the

large firm results is not likely to be due to small sample

size issues in this specificationAnother important dimension over which we might

expect the impact of assistance to vary is whether firms

had previously conducted RampD or not (Fig 4) The

ability to benefit from some types of funding will be

higher when a firm has already done some RampD previ-

ously as they already have the systems and capability in

place Indeed Project Funding is aimed at firms with

existing RampD capability One would expect these firms

to have conducted some kind of RampD activity previously

Similarly one would expect firms that had not previously

conducted RampD to have lower RampD capability and benefit

more from Capability Building programmes

In this model specification we split the sample into firmsthat have undertaken any RampD in the 2 years before firstreceiving RampD assistance We then compare firms inreceipt of Capability Building andor Project Funding as-sistance with firms receiving no government assistance

We see no significant impact for firms that hadundertaken RampD in the 2 years before receiving theirfirst grant In contrast we do see some impacts for firmsthat had not previously undertaken RampD in employmentgrowth and MFP One of the problems is that we believethat many firms that undertake RampD are under-reportingthis activity in their taxation (IR10) forms that have beenused as a data source for this study see Fabling (2008)This means that there will be an unknown number of firmsin the lsquono prior RampDrsquo group that are actually doing RampDin the 2 years before receiving their first grant and this maybe weakening the impact estimates if firms receivemaximum benefit from starting to do RampD for the firsttime We are reasonably confident that firms in the priorRampD group are correctly identified as doing RampD

3 Results

The main results of this evaluation are presented inTable 3 Firms that receive RampD assistance are higher per-forming than the average New Zealand firm Firms arelarger have higher sales and capital intensity and morelikely to be exporting goods and undertaking RampD evenbefore they seek out RampD assistance Failure to take thisinto account when assessing the impact on firm perform-ance due to assistance will result in biased estimates Wereduce the selection bias by matching firms that havereceived assistance with comparable unassisted firms andby comparing the changes in performance of the assistedand unassisted groups before and after receiving the assist-ance Our method is similar to the current best practicemethods used in recent international studies (WorldBank 2010)

Table 3 Main findings

Capability Building assistance 84 increase in sales in first year after receiving assistance no further increase

47ndash67 increase in employment in the first 3 years

15 increase in MFP after 4 years

Project Funding assistance No impact

Large firmsab 106 drop in LP after 3 years

57 drop in MFP during approval year

Small firmsab 5ndash125 growth in employment in the first 4 years

20 increase in LP after 4 years

225 increase in MFP after 4 years

Firms that had undertaken RampDb No impact

Firms that had not previously undertaken RampDb 84 increase in sales in first year after assistance no further increase

4ndash6 increase in employment in the first 2 years after assistance

13 increase in MFP after 4 years

aSmall firms lt62 employeesbCapability and Project funding are pooled

Figure 4 Model 3mdashVariations in impact by previous RampDactivity

304 M Morris and O J Herrmann

Our main model looks at the impact of CapabilityBuilding and Project Funding separately to distinguishwhether impacts depend on the type of assistanceprovided to a firm We see that they do Firms thatreceive Capability Building assistance show significantlyhigher employment growth compared with matched un-assisted firms Most of this growth occurs at the start ofRampD assistance and then grows only slightly after that till3 years following first receiving assistance Still withCapability Building we see a short-term impact on salesand we infer a positive impact on value added because LPdoes not become negative although labour has increasedHowever our most encouraging result is the impact onMFP 4 years following first assistance because this is anultimate outcome for government assistance If this impactis due to firms using resources more efficiently or adoptingbetter business strategies andor practices then we shouldexpect to see the impact continue to be positive at longerlags

In contrast there are no impacts for Project Fundingeven on intermediate outcomes We found thiscounterintuitive because we know that Project Fundinginvolves larger dollar amounts compared with CapabilityBuilding To understand this result better we pooled bothtypes of assistance and examined the influence of firm sizeand prior RampD activity on the results We only foundimpacts for small firms and firms that had not undertakenRampD 2 years before receiving their first assistance We sawno positive impacts for large firms and no positive impactsfor prior RampD performers Our results show that Tech NZhas a significant positive impact when it is targeted at firmsthat are building capability that are small and that havenot previously undertaken RampD

A lack of significant impact on participants in a govern-ment programme does not necessarily mean that govern-ment money is wasted There is potential for spilloverbenefits even in cases where public programmes have anaverage negative impact on recipients For example un-assisted firms may observe the failure of a government-funded RampD project and decide not to undertake a simi-larly risky project themselves possibly resulting in netsavings to the New Zealand economy This is unlikely tobe occurring here given the overwhelming positivefindings from previous evaluations of Tech NZ Anotherargument could be that RampD projects have been so suc-cessful that spillovers have occurred immediately so thatour group of matched control firms already includes firmsthat have benefited by knowledge We find this unlikelyOne concern is that firms that benefit in an indirect mannerfrom other publicly funded projects eg when firms areengaged in partnerships with Crown Research Institutes ofuniversities have not been identified If these firms areincluded in the matched control group then we will under-estimate the impact of direct assistance However thetreated firm is also likely to include some of these indirectlyassisted firms and so the direction of bias is unclear Also

the number of unassisted firms is large we think it isunlikely that the indirectly assisted firms could skew theresults so that all coefficients for Project Funding for largefirms and prior RampD performers are insignificantHowever it is important that we attempt to address thisdeficiency in the LBD by including information aboutfirms that are known to be linked with public providers

Disclaimer

The opinions findings recommendations and conclusionsexpressed in this report are those of the author(s) TheMinistry of Environment the Ministry of BusinessInnovation and Employment and Statistics NZ take noresponsibility for any omissions or errors in the informa-tion contained here

Access to the data used in this study was provided byStatistics NZ in accordance with security and confidenti-ality provisions of the Statistics Act 1975 Only peopleauthorized by the Statistics Act 1975 are allowed to seedata about a particular business or organization Theresults in this article have been confidentialized toprotect individual businesses from identification

The results are based in part on tax data supplied byInland Revenue to Statistics NZ under the TaxAdministration Act 1994 This tax data must be usedonly for statistical purposes and no individual informa-tion must be published or disclosed in any other form orprovided to Inland Revenue for administrative or regula-tory purposes Any person who has had access to the unit-record data has certified that they have been shown haveread and have understood Section 81 of the TaxAdministration Act 1994 which relates to privacy andconfidentiality Any discussion of data limitations orweaknesses is not related to the datarsquos ability to supportInland Revenuersquos core operational requirements

Statistics NZ protocols were applied to the data sourcedfrom the Ministry of Social Development the NewZealand Customs Service the Ministry of Science andInnovation New Zealand Trade and Enterprise and TePuni Kokiri Any discussion of data limitations is notrelated to the datarsquos ability to support these governmentagenciesrsquo core operational requirements

Any table or other material in this report may bereproduced and published without further licenceprovided that it does not purport to be published undergovernment authority and that acknowledgement is madeof this source

Notes

1 The Longitudial Busiess Database contains highquality and comprehensive firm-level data from 2000to 2008

Beyond surveys 305

2 RampD is only one of many inputs into innovationOther innovating activities include learning-by-doingand work reorganization

3 The numbers of firms in the small and large group areequal before matching however many of the smallfirms did not have data for some of the matchingvariables and were dropped in the matching model

References

Bartle D and Morris M (2010) lsquoEvaluating the Impacts ofGovernment Business Assistance Programmes Approachesto Testing Additionalityrsquo Research Evaluation 194 275ndash80

Caliendo M and Kopeinig S (2008) lsquoSome Practical Guidancefor the Implementation of Propensity Score MatchingrsquoJournal of Economic Surveys 221 31ndash72

Coombs R and Georghiou L (2002) lsquoResearch andDevelopment A New ldquoIndustrial Ecologyrdquo rsquo Science2965567 471

de Beer Y Greet P and Morris M (2010) BusinessParticipation in Government Assistance ProgrammesWellington Statistics New Zealand

Fabling R (2008) lsquoWill a BERD Model Fly EstimatingAggregate RampD Expenditure Using a Micro Modelrsquo PaperPrepared for ESAMrsquo08 Wellington

mdashmdash (2009) lsquoA Rough Guide to New Zealandrsquos LongitudinalDatabasersquo Global COE HiStat Discussion Paper Series 103Tokyo Institute of Economic Research HitotsubashiUniversity

Fabling R and Grimes A (2009) lsquoThe Suite Smell of SuccessComplementary Personnel Practices and Firm PerformancersquoReserve Bank of New Zealand Discussion Paper DP200913

Fabling R and Sanderson L (2010) lsquoExporting andPerformance Market Entry Expansion and DestinationCharacteristicsrsquo Reserve Bank of New Zealand DiscussionPapers DP201007

Hall B H Mairesse J and Mohnen P (2010) lsquoMeasuring theReturns to RampDrsquo UNU-MERIT Working Paper Series 006CIRANO - Scientific Publications 2010s-02 Available atSSRN httpssrncomabstract=1566948

Hall BH and Ziedonis R (2001) lsquoThe Patent ParadoxRevisited An Empirical Study of Patenting in the USSemiconductor Industry 1979-1995rsquo Rand Journal ofEconomics 321 101ndash28

Heckman J Ichimura H Smith J and Todd P (1998)lsquoCharacterizing Selection Bias Using Experimental DatarsquoEconometrica 665 1017ndash98

Heckman JJ Ichimura H and Todd PE (1998) lsquoMatchingas an Econometric Evaluation Estimator Evidence fromEvaluating a Job Training Programmersquo Review of EconomicStudies 644 605ndash54

MED (2009) lsquoEvaluation of the Growth Services RangeStatistical Analysis Using Firm-Based Performance DatarsquoMinistry of Economic Development

Rosenbaum P and Rubin D (1983) lsquoThe Central Role of thePropensity Score in Observational Studies for Causal EffectsrsquoBiometrika 701 41ndash55

World Bank (2010) Impact Evaluation of SME Programsin Latin America and Caribbean Lopez-Acevedo G andTan H (eds) The World Bank

306 M Morris and O J Herrmann