Evaluation of business start-up policies by Jonathan Potter, Senior Economist, OECD LEED Programme

Evaluation of Business Start-up Policies

Jonathan Potter

Senior Economist

OECD Centre for SMEs, Entrepreneurship & Local Development (CFE)

jonathan.potter@oecd.org

www.oecd.org/cfe

OECD-European Commission

Capacity-building seminar

‘Financing Business Start-up by Disadvantaged and

Under-Represented Groups’, Trento, 27-29 June 2012

Presentation structure

1. Principles of evaluation

2. Evaluation approaches

– logical framework

– indicators

– qualitative and quantitative evaluation

3. Conclusions

1.1 What is evaluation?

Papaconstantinou and Polt (1997):

“Evaluation refers to a process that seeks to determine as systematically and objectively as possible the relevance, efficiency and effectiveness of an activity in terms of its objectives”

Purpose is to provide decision makers with the best information to answer crucial questions . . .

1.2 Evaluation questions

• Is the activity relevant?

• Is it achieving its objectives?

• Is it cost effective and proportional to what it seeks to achieve?

• Should the activity be continued or terminated?

• If continued, how can it be improved?

1.3 Assessment criteria

Criterion Explanation

Appropriateness (relevance) Extent to which the programme objectives are relevant to the needs of the economy, and address a market failure or social inequity

Effectiveness Extent to which the activity is achieving the programme objectives, net of what would have happened in the absence of the initiative

Efficiency Extent to which costs are minimised in achieving the programme’s objectives

Sustainability To what extent can benefits be expected to last after the intervention is completed

Process management Extent to which the programme is well managed

1.4 The counterfactual

Outputs, e.g.

number of

businesses

started by

the target

group

Time

Baseline Project

Do nothing

option

Assistance

option

Additionality

What would happen without the programme

Our interest is in the difference between the assisted output and the non-assisted output

Non-

additionality

1.5 Evaluation in the policy cycle

Definition of rationale for

policy

Appraisal of policy

options

Implementation of programmes

and projects

Monitoring of progress

Evaluation of results

(benefits – costs)

Feedback

2.1 The logical framework – concept

• Analytical planning tool to clarify causal relations in a policy intervention

• Operational chain of activities, immediate results and subsequent impacts

• Can be used at programme level and at project level

• Can be used for strategy design and for monitoring and evaluation of projects

2.2 Logic framework – microfinance and business support

project

Resources /inputs

E.g. public investment in guarantees of

non-performing loans; fund

administration cost; public

cost of business support

E.g. numbers of loans

provided; hours of

mentoring provided; new contacts with

banks and business support agencies

Activities Outputs

E.g. number of new firms created by

unemployed people;

increased skills;

improved business

proposals; new relationships with banks

Outcomes

E.g. reduction in number of participants

who are unemployed;

increase in number of

starts; increase in survival of

starts; increased lending to

target group

Impacts

E.g. reduction in

unemployment in the target

group; increased

entrepreneur-ship rate

2.3 EQUAL Business Creation Theme Logic Model

2.4 Indicators

Type Examples

Inputs Funds allocated Total budget Funds allocated as % of budget

Activities Number of people contacted Number of people having received services Number of proposals for the creation of new businesses Number of advisors % of beneficiaries belonging to target groups

Outputs/outcomes Attitudes to entrepreneurship Business start rate Survival rate of enterprises Reduction in unemployment rate

Other Satisfaction of beneficiaries with the programme Satisfaction of managers with the programme

2.5 Methods for evaluation

Qualitative approach

• Based on stakeholder and beneficiary interviews

• Deep questioning on processes and changes in decisions and behaviour

• Strength is that it explores processes and engages participants in policy learning

Quantitative approach

• Compares performance of assisted and matched firms and individuals

• Accounts for selection bias

• Strength is that is gives robust impact estimates

12

3. Conclusions

• Evaluation is critical to policy success but is not yet properly integrated in policy process

• At this stage of the EU policy process the emphasis is on ex ante evaluation, implying:

– development of logical frameworks

– Identification of indicators to measure, targets to reach and creation of a monitoring programme

• Once activities have been implemented, will need to consider the nature of evaluations to commission; may need some data collection now