Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Project Presentations
Target for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
How we’re doing this today
• In pairs: two presentations of 5 minutes, followed by 15 minute Q&A
• There is a lot you would want to talk about
How we’re doing this today
• but resist the temptation and deliver
How we’re doing this today
Zambia – Tunisia
Mexico – Vietnam
Bangladesh – Greece
Kenya – Senegal
Argentina – Colombia
Brazil – Peru
Ethiopia - Kosovo
Intermediation and Matching Grants for Agribusinesses in Zambia
Target for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
Intervention(s) to be Evaluated
Executed by the Ministry of Commerce, Trade and Industry
Overall goals: Increased market linkages and firm growth in agribusiness.
Components: Market Linkages in Agribusiness (US$28.4 million)
• Subcomponent 1a. Building Productive Alliances in Zambia (US$16.7 million)
(a) matching grant investments in Productive Organizations (POs) (approximately US$9.7 million); (b) facilitation of and capacity building in POs and PAs (approximately US$3 million);
• Subcomponent 1b. SME Supplier Development Program (US$11.7 million)Invest in MarketConnect (US$6 million) (training and BDS program) and Business Linkage Fund (US$5.7 million – matching grants focusing on product upgrade/standard development equipment/ alternative energy) (
Scope
• Productive Alliances in Zambia – FarmersConnect: - 180 PAs (by end year 5), 4500 emerging farmers, PAs need to include minimum of 10 farmers, each farmer MG capped at 3000 USD, 30 % counterpart financing required
• MarketConnect- expected to reach a total of 300 SMEs, of which 140 SMEs expected to participate in the Business Linkage Fund, 10,000 – 100,000 royalty bearing matching grants
• Minimum eligibility criteria and definitions for emerging farmers and growth-oriented SMEs
• Reach – covers five provinces which include the highest density of poverty and agribusiness
Theory of Change (Productive Alliances in Agribusiness/Market Connect for SMEs)
LONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS
Aggregation of farmers
into POs / activation of
existing coops
ACTIVITIES
Information campaign
about structure of
program and
selection processHigher revenue,
profitability,
productivityIntermediary Service
Provider providing
coordination and
planning T.A.
Handholding PA/SME
to allow absorption
and assistance in
implementation
Business plan formed for
PAs/SMEs
Invest in PO’s to form
productive PA’s (High
potential POs matched
with sellers, )
business plans developed
and submitted for reviewImproved market
access, reduced
transaction challenges
MEDIUM-TERM OUTCOMES
Contract Fulfilled,
sustained over medium-
run
Technical Review of
proposals for PA’s
and SME’s
Formal agreement
between buyers and
PO’s/SMEs (Contracts)
Behavioral change:
investment, managerial
practices
Reduction in
costs/Increase in
markup
Improved reputation
amongst buyers and
retailersPA’s receive MG based
on achievement of first
sets of milestones
SMEs put in place
Marketing strategy*
+
Subset apply for and
receive MG
Changes in product mix*
Increased Sales,
employment*
Contract specific
investment aimed at
improving productivity
Matching Grant to
elegible PA’s & SMEs
Eligible PA’s & SMEs
selected to receive TA &
matching grant *outcomes formarket connect
Impact Evaluation Questions• Overall Question: What is the impact of intermediation and providing matching grants on income,
productivity, firm size and transaction challenges? • Conditional on being matched to a buyer, what is the effect of receiving support in developing business plans?
(T1)• Conditional on receiving business plan support, what is the effect of receiving a matching grant? (T2)
(Two distinct interventions: 1) Matching farmers in POs to buyers (Productive Alliances Program) and 2) Matching SMEs to buyers (Market Connect Program)
(Intermediaries do 3 functions: matchmaking between PO and buyer, support to development of business plan, and support to execution of business p (financed by the MG)
• Main indicators and data• Key indicators: productivity, sales, income, employment, reputation (e.g. scores/reviews), transaction challenges
• Data availability: admin data, project-specific surveys, Zambia Enterprise Survey, Zambia Business Survey
• Upstream Challenges and Delivery Mechanisms:
• Does intensive handholding lead to a higher probability of bringing farmers together to form a Productive Organization? (Randomize villages for intensive PO-outreach)
• Conditional on forming a productive organization, what is the effect of being matched with a buyer? (Randomize buyer to match to PO)
• Conditional on being selected to receive a Matching Grant, what is the optimal subsidy level to induce take-up? (Needs further work)
Methodology
Total sample size
540 eligible PAs
Control Group
[180 PAs]
T2
[180 PAs]
[Business Plan +
Matching Grant]
T1
[180 PAs]
[Business Plan ONLY]
Treatment Group
[360 PAs]
Challenges
• Take Up• Farmers/Producer Organizations (testing handholding intensity, information
channels)• Buyers/Producer Alliances (intermediary intensity and quality)
• Attrition• POs not able to provide quantity/quality as promised, buyer pulls out, PA
collapses (intermediary intensity and quality)
• Contamination (examining spillover measures)
• Implementation• Producers buying into counterpart financing• Sufficient Governance structures of POs• High intensity handholding during implementation/disbursement
Impact Evaluation Timeline
Project Effectiveness
Mar 2017
Implementation
June 2017
Baseline
Sep 2017
Follow up 1
May 2018
Follow up 2 -4
May 2019-21
Final dissemination
MMM YYYY
• Operations Approval Process
• Information Campaign - Apr
• Intermediaries Hired – June
• First Sets of Proposals – July
• Proposal Deadline – August
• Business Plan Support –September/October
• 1st milestone disbursed -November
• Cleaned dataset
• Data analysis• Client
workshop• Technical
paper• Policy note
• Cleaned dataset
• Data analysis
• Technical paper
• Policy note• Client
workshop
Thank you!
How Effective is Productive Alliance to help grow the Small Scale Producers in Tunisia
Target for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
What is a Productive Alliance
Intervention(s) to be Evaluated
• Context• Agencies involved: Ministry of Agriculture, Ministry of Industry, Ministry of Commerce
and Trade• Key overall goals: Improve small-scale producers’ income and job creation• Components: create productive alliance for small-scale producers and buyers
• Scope: • 20 productive alliance will be formed in 10 villages. Each alliance has about 50 farmers.• On average, 200,000 USD matching grants will be offered to each alliance. • Buyers offer suppliers’ contracts and product specific training to ensure product
quality/variety
• Only small producers with irrigated land less than 2 hectares can participate
• Reach: the first phase of the program will be carried out in 10 villages, the program will be expanded to more villages in the future.
Theory of Change
LONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS
Application for joining
productive alliance
Number and value of
contracts, price of
inputs, product quality,
product variety,
investment, PA-jobs
ACTIVITIES
Information session
on benefits of
participating in
productive alliance
Employment
Randomly select
applicants to join the
productive alliance
Form partial
productive alliance
(productive alliance
without matching
grant) in selected
control villages
Training/Capacity
Development -
knowledge on product
quality, market
information, marketing
strategies, management
practices etc.
Form productive
alliance
Form partial productive
alliance
MEDIUM-TERM OUTCOMES
Productivity, profit
Impact Evaluation Questions
• Overall What is the impact of Productive Alliance on profitability, growth, and productivity of Small-scale producers• Specific question (arm 1): what’s the impact of joining productive alliance on
farmers’ market access, income, productivity, and employment?
• Specific question (arm 2): what’s the impact of joining partial productive alliance (networking with buyers but infrastructure investment and grants) on farmers’ market access, income, productivity, and employment ?
• Main indicators and data• Main indicators: market knowledge, sales, profit, productivity, jobs
• Data: surveys of small-scale producers, administrative farm census data from the Ministry of Agriculture
Methodology
• Small-scale producers will be randomly selected to participate in the Productive Alliance in 10 villages
• Sample villages will be randomly selected (out of 100 eligible villages) for “partial” productive alliance
• Small scale producers in treated villages for “partial productive alliance will be randomly selected to participate in partial productive alliance
• Surveys and admin data will be used for evaluation purpose
Total sample size
3000 small-scale
producers in 30 villages
Control Group
[1000 farmers in 10
villages]
[None]
Arm 2
[1000 farmers - 20
productive alliance in 10
villages and 20 buyers]
[Partial productive
alliance]
Arm 1
[1000 farmers - 20
productive alliance in 10
villages and 20 buyers]
[Productive alliance]
Challenges
• Challenge 1: Take-up of producers and buyers• Lack of trust, lack of knowledge of the benefits of productive alliance • Mitigation: face-to-face meetings, information session.
• Challenge 2: Measuring spillovers• Non-participating farmers live in the same village with participating farmers• Mitigation: compare non-participating farmers in treated villages with farmers in control
villages.
• Challenge 3: Attrition• Farmers/buyers exit the productive alliance over time• Mitigation: state clear in the contract about the punishment of exiting.
• Challenge 4: Decrease in market demand• Economic crisis • Mitigation: assist in improving product quality/variety; find new markets.
Impact Evaluation Timeline
Concept
Aug 2017
Baseline
Oct-Dec 2017
Implementation
Jan 2018
Follow up 1
Jan 2020
Follow up 2
Jan 2021
Final dissemination
Sep 2021
• Funding application
• Approval process
• Cleaned dataset
• Data analysis• Client
workshop
• Monitoring report
• Lessons note
• Cleaned dataset
• Data analysis• Client
workshop• Technical
paper• Policy note
• Cleaned dataset
• Data analysis
• Technical paper
• Policy note• Client
workshop
Thank you!
Discussion
Connecting Lead Firms to Local Suppliers in Mexico
Target for Growth: Impact Evaluation WorkshopMexico City
February 27th- March 3rd, 2017
Background
•Context: Four new SEZs in Mexico (Chiapas, Veracruz, Michoacan, Oaxaca)•Motivation: A major objective is to generate spillovers to local suppliers in the “zona de influencia” •Proof of Concept: Design an online marketplace to foster linkages between lead firms and potential domestic suppliers, determine the value of these relationships •Priors:
•Finding local suppliers is costly for lead firms•The quality and specifications of locally produced goods may not initially satisfy lead firms’ sourcing needs
Rubén
Theory of ChangeLONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS
General training
for all firms,
intense
mentorship from
lead firms
ACTIVITIES AND INPUTS
Capacity
building with
local firms:
training,
diagnostics,
mentoring
Gather
information:
surveys, photos,
reliability
rankings
Use platform to
determine
matching,
prices
More eligible firms,
more initial
contracts created
Matching system
for local firms and
lead firms,
arrange
relationships
MEDIUM-TERM OUTCOMES
Transfer of
expertise to local
firms Greater overall use
of local firms:
product line
diversification
Arrange contracts
and financial
support for local
firms
Repeated
contracting and
relational
investment
Local firms
increase future
productivity,
investment
Increase lead firms'
interest and lower
their risk aversion
in working with new
suppliers
Lead firms switch
from importing to
local sourcing
Rafael/
Priam
Impact Evaluation Questions Rafael/
Priam
•What is the impact of training/support on lead firms' demand for contracts/relationships/links with local firms?•How can intensive training/mentorship increase the likelihood of link formation?•How does differential information about potential partners impact link formation?•What is the impact of price discounts/assistance on continued lead firm-supplier linkages?
Methodology
•Local suppliers randomly sampled from SEZs, lead firms are in negotiations
Treatment:
High-intensity training,
1000 firms
Control:
Low-intensity training,
1000 firms
Partial
information:
500 firms
Full information,
linkage support:
500 firms
Full information:
500 firms
Training Phase:
Matching Phase:
Full information:
500 firms
Challenges
•Challenge 1: Lead firms might not want to work with local firmslA feature, not a flaw?lStickiness of existing relationships
•Challenge 2: Too few lead firms or local suppliers•Challenge 3: Firms contract off the platform around us
lThey may renegotiate the terms we set up or leave the platform•Challenge 4: Platform implementation
lPartnering with an existing platform vs competing with them•Challenge 5: Financial incentives
lSubsidies are controversial; insurance may be hard to implement•Challenge 6: Underbidding and default
lFirms might put in unrealistic bids to win, then fail to deliver
Impact Evaluation Timeline
ConceptAug 2017
BaselineJanuary 2018
TrainingApril 2018
MatchingJune 2018
EndlineDec2019
DisseminationDec 2020
•Funding application•Approval process
•Survey, photograph, and screen local firms•Survey lead firms •Develop platform
•Conclude low-cost and high-cost training•Solicit needs from lead firms, proposals from suppliers
•Begin arranging matches on platform•Evaluate impact of training on link formation
•Survey lead and local firms on key outcomes•Evaluate impact on firm and relational outcomes
•Academic papers•Policy note•Demand models for SEZ prediction•Transfer platform to government
Thank you!
Vietnam: facilitating FDI-SME linkages
Target for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
Intervention(s) to be Evaluated
• Project objectives:• To expand the integration of local SMEs into GVCs and increase overall domestic
value added
• Participants: • Buyers: FDI firms in targeted value chains (automotive & electronics)
• 15-20 lead firms and potentially 100+ tier-1 or tier-2 suppliers
• Sellers: local SMEs in relevant upstream sectors (plastic, rubber, metal processing)• Around 400 firms with potential to participate
• Implementing agency: Ministry of Industry & Trade, Supporting Industry Enterprise Development Center
Intervention(s) to be Evaluated
• Interventions:• Online supplier database
• Scoring firms on potential to be successful suppliers
• Coaching program, in conjunction with demand side, for improving success as suppliers
Theory of Change
• Develop the chain of effects from interventions to be evaluated to outcomes
• Discuss potential unintended effects and assumptions
LONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS
Improved quality of
informationTransaction cost (time
spent searching for
potential suppliers)
Usage, WTP
ACTIVITIES
Online database
Scoring of firms
Growth (sales,
employment)
Productivity
Investment
Spillovers
Supplier development
programImprove performance
score of SMEs
Number of firms trained
Adoption of improved
production management
by SMEs
MEDIUM-TERM OUTCOMES
Increased number of
contracts and sales to
FDI
Impact Evaluation Questions• Does the program increase linkages between SMEs and FDI firms?
• What is the effect of providing scoring information in the online database?
• What is the effect of the supplier development program?
• Main indicators and data• Number & value of contracts
• Firm outcomes: sales, employment, profits, productivity, investment, spillovers
• Data:• SIDEC & Ho Chi Minh supplier databases: 300-400 suppliers with potential to join
• Administrative data from the SDP + survey data
• Existing and potential FDI buyers (lead firms & tier-1 suppliers of lead firms): Foreign Investment Agencies + provincial Planning & Investment departments + IZ Authorities
Methodology
Total sample size:
400 local firms with
potential to become
suppliers
Control Group
[175]
Arm 2: Providing
“readiness score” in
online database
[200]
Arm 1: Coaching
program to meet buyer’s
standards
[50]
Challenges
• Discuss potential challenges/risks and how to tackle/mitigate them• Power (sample size, treatment group size, non-compliance, general
participation)• High-frequency data collection
• Spillovers (+/-)• Collect network data to measure effects (market competition, firm interactions)
• Implementation failures:• In baseline: build on rationale for theory of change & identify potential obstacles
Impact Evaluation Timeline
Concept
Aug 2017
Baseline rounds
Fall 2017
Implementation
Early 2018- June 2021
Follow up rounds
Throughout & after
implementation
Final dissemination
Dec 2021
Thank you!
Discussion
One-Stop Shop services for private investors in Bangladesh
Target for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
Intervention(s) to be Evaluated
• We will evaluate the implementation of a One-Stop Shop (OSS) for investor services and process simplification in Bangladesh.• Agencies involved: Bangladesh Investment Development Authority (BIDA), in
collaboration with line agencies.• OSS will simplify the process for both foreign and domestic investors to obtain entry
and operational licenses for investment projects in Bangladesh.
• Scope: BIDA receives approximately 10,000 license applications per year. The list of recent applicants for investment projects will form the basis of our sampling frame.
• Eligibility: The current bill focuses on all foreign and domestic investors.
• Reach: The OSS will serve all sectors and regions of the country through a combination of main and regional offices, and an online platform.
Impact Evaluation Questions
• Overall: What is the impact of the process simplification and lowered transaction costs on the growth of new private investment?• Does the OSS reduce time and cost of private investment entry?• Does the OSS improve satisfaction with the process of private investment entry?
• Main indicators and data• Indicators: Number of applications submitted to OSS vs. “status quo” process,
satisfaction with process, time to approval, share of applications completed/approved, total investment proposed, total investment realized, incidence of corruption
• Data availability: BIDA investor tracking systems/database, companies register, Central Bank survey of foreign investors, feedback surveys, internal databases held by line agencies and OSS, baseline/endline surveys by IE team
Methodology
• Once the OSS is officially established and launched, we will implement a randomized encouragement design to induce exogenous variation in the use of OSS.
• (Institutional constraints will not allow restricted-access pilots or geographic staggering.)
• We will base the sampling frame on the former business investment license applicants to BIDA (~10,000 annually, use most recent year as initial frame; additional years as available).
• We will use existing administrative data (applicant/process files) + baseline and endline survey.
Total sample size
Estimated maximum
~10,000
Control Group
33%
No incentive or
matching service
Arm 2
33%
Encouragement/incentiv
e to use OSS
+
BIDA buyer/supplier
matching service
Arm 1
33%
Encouragement/incentiv
e to use OSS
Challenges• Four common examples:
• Challenge 1: Take up• Investors and/or intermediaries may prefer the well-established process with which they are familiar (or, lack
confidence in new OSS process; potential perception by investors of likelihood of increased effectiveness of OSS). • Possible encouragement incentives may be too small for relatively large investors.• We will have further consultation with government and investors/firm owners to determine appropriate level of
incentive.• We will allow for an expanded study horizon within which to measure take-up.
• Challenge 2: Contamination of control group.• We believe this to be unlikely based on sampling frame.• Will frame the intervention as a randomly-assigned inaugural promotion which should reduce likelihood feelings
of unfairness.
• Challenge 3: Implementation failures. Possible causes:• Resistance of line agencies to participate in OSS or no line agency process simplification. • Intermediaries may block take-up of OSS. • Strong commitment of client to ensure establishment of OSS and IE.
• Challenge 4: Confounding effect of incentive• Instead of a cash incentive, we will use credit for consumer goods so as to not affect investor’s business/financial
situation directly.
• Challenge 5: Proper measurement of corruption• Will be based on self-reports in baseline/endline surveys.
Impact Evaluation Timeline
Concept
Aug 2017
Pilot
by September 2017
Implementation: OSS online
by June 2018
Follow up 1
by June 2019
Follow up 2
by June 2020
Final dissemination
by June 2021
• Funding application
• Approval process
• Sampling frame available
• Client workshop
• Pilot
• Baseline survey
• Assignment/incentive offer
• Intervention monitoring
• Obtain post-intervention admin. data
• Midpoint survey + report
• Client workshop
• Endline survey• Obtain post-
intervention admin. data
• Data analysis• Draft technical
paper
• Technical paper
• Client workshop
• Policy note
Theory of Change
• Develop the chain of effects from interventions to be evaluated to outcomes
• Discuss potential unintended effects and assumptions
LONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS
1) Establishment of One-
Stop Shop (OSS) services
• Minimum five (5) line
agencies linked to OSS
• Necessary institutional
structure to support
OSS
2) Information widely
available on online portal
3) New investment
process simplified
4) Reduce human
interaction
• Use of OSS for new
investment projects
• reduced time and cost
for regulatory approval
for private investment
• Increased investor
satisfaction with
approval process
• Increase in applications
• Increase in share of
project applications
completed/approved
ACTIVITIES
• Formulation and
enactment of OSS
law
• Development of
rules and
regulations
• Integration of
services and
process
simplification with
line agencies
• Establishment of
online OSS portal
• Training of
involved staff
• Strategic
communications to
private investors
• Increased job creation
MEDIUM-TERM OUTCOMES
• Increased private
investment
• Increased business
entry
• Reduced incidence of
corruption related to
processes
• Increase in economic
diversification
Thank you!
Investment Licensing Reform in Greece
Target for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
• Δ/νση Αδειοδότησης Επιχειρήσεων & Επιχειρηματικών Πάρκων, ΓΓΒ/ΥΠΟΑΝ (Ιαν' 2017)
Intervention(s) to be Evaluated
• Describe the interventions to be evaluated
Optimize the inspection system to increase efficiency and maintain (or increase) safety
Reduce the time (and cost) to notification for businesses
What are key overall goals? What are the components?
• Goals: Improve the business environment, improve efficiency of the inspection system (optimize resources), preserve or improve public safety
• Components: Assist businesses during the initial notification process, vary the inspection schedule for checking for compliance
• Scope
Target sectors: Food and Beverage production, Shops with hygiene requirements (restaurants, hairdresser, night clubs, etc)
• Eligibility: Both new and old firms (excluding very high risk firms)
• Reach: Pilot in Attica Region (1 of 13 in Greece), main project cross country
Activities Intermediate outcomes
Improve knowledge of firmsImprove time to notificationReduce cost to notificationImprove quality of information provided
Improved economic
activity (job
creation)
Increased number of
firms created
Components/inputs
1
Simplified regulatory framework
for business
2
Improved
inspection
system to
check for
compliance
with rules,
and enforce
sanctions for
non-
compliance
Long-term
outcomesOutputs
Inspection
Improved Inspection
quality and efficiency
• Change license to
notification
• Reduce approvals
• Provide information on
requirements
• Develop inspection
probability system by risk
and time
• Inform some businesses of
probability risk
• Develop inspection quality
check/system
• Develop information,
guidance to support
compliance
• Notification system
including information on
key steps
• No. of inspections
• No. of quality checks
conducted
• Information/guidance
support delivered
• Sanctions enforced (%)
Improved firms’
compliance with safety
standards, reduced
safety violations
Improve safety /
mitigate safety risks
3
Improved information
to/from consumers
a) Scorecard system
b) Digital feedback
Theory of Change
Unify penalty system
Develop scoring system from checklistDevelop platform for customer feedback
• Scoring system displayed
in businesses
Notification System
Framework Law
Impact Evaluation Questions
• How can we optimize inspection system to increase efficiency and maintain (or increase) safety?• How does safety and efficiency vary with inspection probability?
• How does safety and efficiency vary depending on disclosing inspection probabilities?
• How does safety and efficiency vary depending on timing of inspection?
• How can we reduce the cost of and time to notification for businesses?• How does information provided to businesses at registration affect
notification compliance and cost?
Main indicators and data
• Outcome Data for Inspections• Compliance with safety regulation as determined at inspection (0 or 1)
• Safety score from inspection checklist ( continuous)
• Quality of inspections by shadow inspectors (effort and corruption)
• Consumer reporting of suspected violations
• Cost of inspections
• Firm reported profits
• Outcome Data for Notifications• Time to notification
• Quality of information provided
• Cost to businesses
Other Impact Evaluation Questions we were interested in (and will look at later)• Does advertising the simplified notification system lead to an increase
in the number of new businesses?
• Does displaying a public rating system (based on compliance) improve safety and affect profits for firms?
Methodology - Inspections
Total sample size
11,000
Control Group
Inspect based on current
risk probability
calculations (status quo)
Arm 2
Unknown probability of
inspection
(Vary probability of
inspection)
Arm 1
Known probability of
inspection
(Vary probability of
inspection)
Information treatment for notification (for half of all
above firms)
Stratify by high risk/low
risk
Challenges
• Challenge 1: Compliance with the inspection schedule• Challenge 2: Measuring spillovers
Impact Evaluation Timeline
Concept
Aug 2017
Baseline
June-Nov 2017
Implementation
Nov 2017 – May 2018
Follow up 1
June 2018-Nov 2018
Final dissemination
Dec 2018
• Funding application
• Approval process
• Cleaned dataset
• Data analysis
• Monitoring report
• Lessons note
• Cleaned dataset
• Data analysis• Client
workshop• Technical
paper• Policy note
• Technical paper
• Policy note• Client
workshop
Thank you!
Discussion
Business Plan Competition in KenyaTarget for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
Intervention(s) to be Evaluated
• Program: Kenya MSEA’s Business Plan Competition with the creating jobs for targeted youth*
• Scope: 7,500-10,000 applicants; 3,000 screened for preparing business plans; 1,500 support in Business Plan preparation; 750 receive grants (500 through screening process)
• Eligibility: Young entrepreneur up to 35 years old with a registered youth-owned business** or intending to register one; Kenyan
• Reach: National level
*youth between 18 and 29 years of age, who are without jobs and have experienced extended spells of unemployment or who are currently working in vulnerable jobs (working on their own, a contributing family worker, or working for wages in a household enterprise with fewer than 10 workers. These are mostly informal workers). The level of education of targeted beneficiaries will be up to Form 4.
**75 percent owned by individuals up to 35 years old
Theory of Change
LONG-TERM OUTCOMES
(12-60 months)
SHORT-TERM OUTCOMES
(1~3 months)OUTPUTS
• 750 business ideas
with potential for
growth identified
• Businesses created/in
operation
ACTIVITIES
• Application &
Selection
• Awareness
& Call
• Application
• Initial
Screen
• Review of
submitted
business
plans
• Sustained targeted
youth employment
• Targeted youth with
access to continued
skills training.
• Support for
Business Plan
Preparation
• Grants of $9,000
• Grants of $36,000
- Business grants to new
and existing
businesses.
- 500 businesses with
$9k
250 businesses with
$36k
• Increased business
profits
• Increased business
investment
MEDIUM-TERM OUTCOMES
(3 -12 months)
• Greater number
employees hired
• Higher profits
• Businesses’ with
increased access to
finance
• Businesses’ with
increased access to
networks
• Businesses’ with
increased access to
additional skills
• 1,500 youth
entrepreneurs trained
to create a business
plan
• Refined business ideas
with high potential.
• High-impact
entrepreneurs with
business plans
• Greater number of
target youth employed
• Higher earnings for
targeted youth
• Increased quality of
employment.
Impact Evaluation Questions
• What is the impact of a grant to high growth entrepreneurs on job creation for targeted youth in the context of a business plan competition?• What is the differential impact of providing a large grant ($36,000) versus a small
grant ($9,000)?
• Does support for business plan preparation increase the likelihood of winning the business plan competition?• And the ultimate success of the businesses? [conditional on power]
• How effective is the business plan competition screening process at selecting firms that will provide job creation for target youth?
• What is the effect of the business plan competition screening process on firm impact?• For those who do not participate in the business plan competition screening process,
what is the effect of a small grant ($9,000) on job creation for target youth?
Impact Evaluation QuestionsOutcome Indicator
Business in operation Self Report (SR), Online check, Physical check
Access to skills SR, Expert Evaluation
Access to finance SR, KRA records, financial due diligence
Network characteristics SR
# employees / employees in target youth SR, Observation, KRA
Proportion full time SR, Audit
Income of workers SR, KRA
Type of employment SR, Audit
Business profits SR, KRA
Business investments SR, Audit
Methodology Wednesday 1
Total
sample size
10,000
5. Selection + $0+ Support
6. Selection + $0
250
3. Selection + $9k + Support
4. Selection + $9k
250
1. Selection + $36k + Support
2. Selection + $36k
250
7. $9k
2508. $0k
250
Eligible
9,000
11. Support
1,500
10. No
Support
2,500
Plan
Submitted
2,000
Screen
Passed
4,000
Winners
750
(225 Support + 225 No Support + 300)
Ineligible
1,000
Screen
Failed
4,500
Losers
1,250
Not
Submitted
2,000
9.
Screening
Sample
8,500
(500)
Groups
Group Selection Support Winner Grant Size
n
1 X X X $36k >75
2 X X $36k >75
3 X X X $9k >75
4 X X $9k >75
5 X X X $0 >75
6 X X $0 >75
7 $9k 250
8 $0 250
9 X ? ? ? 500
10 X ? ? -
11 X X ? ? -
Effect Comparison
Any grant (1+2+3+4) - (5+6)
Grant Size (Large vs. Small) (1+2) - (3+4)
Support (on winning) (10) – (11)
Support (on winner outcomes) (1+3+5) – (2+4+6)
Selection on picking winners (4+6+10+11) – (7+8)
Selection on firm impact (9) – (8)
Grant (cond. no selection) (7) – (8)
Challenges
• Challenge 1: Not enough applicants to the Business Plan competition
• Applicants do not have access to the internet
• Online platform is not operational (downtime, slow, confusing)
• Phishers impersonate the website to collect user information
• Multiple applications by an individual or firm artificially increases the sample size.
• Too difficult to submit the business plan
• Business plans are copies or fake
• Challenge 2: Difficulty in determining eligibility
• Criteria is difficult to measure or verify
• Criteria is too strict
• Verified eligibility reveals that winners were not eligible ex-ante
• Challenge 3: Attrition
• Applicants do not attend the support program
• Non-response from people who do not win the competition
• Challenge 4: Non-compliance of grant recipients
• Grant recipients do not adhere to tranche rules
• Challenge 5: Mistakes in carrying out research implementation
• Process fails to adhere to randomization policy
• Political landscape forces changes to implementation
• Implementing firms fail to carry out project as intended
• Challenge 6: Spillovers
• Collusion among potential candidates
Impact Evaluation Timeline
Concept
Aug 2017
Implementation
Jun 2018
Baseline
Jan 2019
Follow up 1
Sep 2019
Follow up 2
Jan 2021
Final dissemination
Dec 2021
• Funding application
• Approval process
• Monitoring report
• Lessons note
• Cleaned dataset
• Data analysis• Client
workshop
• Cleaned dataset
• Data analysis• Client
workshop• Technical
paper• Policy note
• Cleaned dataset
• Data analysis
• Technical paper
• Policy note• Client
workshop
Impact Evaluation TimelineWednesday 2
Thank you!
Senegal Business Plan competitionTarget for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
Intervention(s) to be Evaluated
• The intervention: Business Plan Competition
➡Which agencies are involved? ADEPME
➡What are key overall goals? To encourage innovative firm creation and/or growth
➡What are the components? Outreach, selection, training, grants, aftercare services
• Scope: 4 rounds; up to 150 awardees per round (500 total); 4-5 days Business Plan Training 1,500 trainees per round (6000 total). Grants range: 5,000-50,000 USD
• Reach: The program will be implemented on a national level
Theory of Change
LONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS
Outreach campaign
completed
Sufficient number of
entrepreneurs applied
Public awareness on
entrepreneurship
increased; motivation of
aspiring entrepreneurs
increased
ACTIVITIES
Outreach &
communication
campaign and selection
process
Job creation
Business plan training
and submission of
business plans
Selection of finalists,
grant funding and
aftercare services and
support
Entrepreneurs utilizing
knowledge, services and
money of the program to
scale their businesses
Selected entrepreneurs
trained
Business plans developed
and submitted
Finalists submissions
evaluated
After-care Services
delivered
Funding disbursed
Revenue increase
MEDIUM-TERM OUTCOMES
Spillover effect
Increased innovation
Survival rate
Impact Evaluation Questions
• Overall: What is the impact of BPC on the innovative firm creation and growth in Senegal?
• Arm 1: Does BPC as currently implemented affect innovative firms creation?
• Arm 2: What is the additional impact of the follow up customized support (aftercare) to the winners on the firms survival rate and growth?
• Arm 3: What share of the program impact on the firms survival and growth comes from public recognition and celebration as “winners”?
• Main indicators and data• Key indicators: firms created, firm survival rate, employment, revenues, innovation• Data available: project data (ERP), administrative data (RCCM, RNEA); impact
evaluation data (new survey, including baseline, midline and end-line surveys)
Outcomes Definition
• Originality, relevance, feasibility (Innovation index to be refined)
• Firms still existing after the 1st and the 3rd year
• Turnover
• Number of employees present by the day of the survey
• Increase of number of applicants among respondents baseline network• Negative effects on competitors
• Index of self-confidence• Impact on marketing expenses
Methodology
800 selected applicants
BPC : 350 WINNERS
BIG CELEBRATION
OF THE WINNERS
NO CELEBRATION
ADDITIONAL AFTERCARE SUPPORT
A B 175
NO AFTERCARE SUPPORT
C D 175
175 175
Control Group
450
• Baseline: at the selection process,
• 2 follow-up surveys
Challenges
• Challenge 1 : Is randomization acceptable by project stakeholders? ➡Mitigation: Additional consultation to explain the benefits and to discuss the
best way to communicate the transparency of the method
• Challenge 2: Take up➡Based on the pilot implementation, the demand is high
• Challenge 3: Contamination➡Phased in implementation
Thank you!
Discussion
Buenas Prácticas exportadoras(Good Exporting Practices)
Target for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
Intervention(s) to be Evaluated
• Describe the interventions to be evaluated• Agencies are involved: AAICI, INTI, STP, Secretary of commerce• Overall goal: improve export performance• Components: Diagnostic of best practices in 7 specific areas (strategy, market
identification and segmentation, design and development, production, communication,distribution and administration) and technical assistance
• The diagnostic and the technical assistance will be performed by specialists in exportpromotion, commercialization, design, quality, and strategy
• Scope: 300 SMEs, budget: USD1.5 million
• Eligibility: SMEs in the food industry, 5 to 250 employees, export experienceor willingness to export
• Reach: national level
Theory of Change
LONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS
Report of results in the
7 areas mentioned
previously
Change in practices
ACTIVITIES
Diagnostic
Export performance
(value, volume)
Employment
Diversification of
exports (number of
products , number of
markets, sophistication
of export markets)Technical assistanceImplementation of a
plan to change practices
MEDIUM-TERM OUTCOMES
Change in practices
Export performance
(value, volume)
Employment
Diversification of
exports (number of
products, number of
markets, sophistication
of export markets)
Impact Evaluation Questions
• Lack of knowledge of good business practices constrains export potential
• Does adoption of good export practices lead to better export performance?
• How to facilitate information of good export practices to SMEs? • providing a diagnostic incentivizes adopting good exports business practices? Or• Do you need to combine both diagnostic + technical assistance
• Main indicators and data• Index of export practices adoption • Exports (value, volume, quantity of export destination, sophistication of market)• Employment
Methodology
• The budget is only enough to perform the diagnostic on 200 firms and the technical assistance on 100 firms
• If the number of firms that express interest in participating exceeds 200, we will randomly assigned 200 firms to receive the diagnostic, and100 firms to receive the intervention, so the remaining for will be assign to a control. Otherwise, we will assign half of the firms to receivethe diagnostic, and half of the firms to receive the diagnostic and the technical assistance
• Two sources of data: administrative records and surveys conducted by the impact evaluation team (3 surveys, a baseline and two follow upstaken by firms in Arm 1 and Arm 2)
Total sample size
300 (or n, if different
from 300)
Control Group
[100, or n-200]
Arm 1
[100]
[Diagnostic]
Arm 2
[100]
[Diagnostic + technical
assistance]
Challenges
• Ensure adequate quality in both implementing the diagnostic and the diagnostic + technical assistance• Training of consultants• The diagnostic has been piloted• Monitoring of the consultants
• Four common examples:• Challenge 1: Take up
• There are multiple strategies (mailing to firms in the export database, information posted on Agency’s website and social media, mailingto members of 50 chambers of commerce, mailing to provincial export promotional agencies database, etc.) to achieve the target takeup of 300 firms
• Challenge 2: Contamination of control group• All other interventions by the government agencies are orthogonal to the treatment (even if they are other interventions, all firms are
targeted independently of treatment status)• Blind process: Consultants surveying firms are not informed on whether the firm they are surveying will be treated or not
• Challenge 3: Implementation failures• Potential non compliance. This will be addressed using an Instrumental Variables approach• Attrition. No problem for administrative outcomes
• Challenge 4: Measuring spillovers• Positive spillovers would arise if some firms can obtain information on exporting practices even if they are not treated. However, this is
unlikely to be the case since the nature of the program is to be “taylor-made”
Impact Evaluation Timeline
Concept
Jan-Feb 2017
Baseline
May – Jul 2017
Implementation
Aug - Jan 2018
Follow up 1
May – Jul 2018
Follow up 2
May – Jul 2019
Final disseminatio
n
Aug-Dic2019
• Cleaned dataset
• Data analysis• Client
workshop
• Monitoring report
• Lessons note
• Cleaned dataset
• Data analysis• Client
workshop• Technical
paper• Policy note
• Cleaned dataset
• Data analysis
• Technical paper
• Policy note• Client
workshop
Team
Impact Evaluation Team• Impact evaluation TTL - Leonardo Iacovone
• Principal investigator(s) - Martin Rossi
• Research analyst(s) - Maximiliano Lauletta
• Field coordinator - (graduate student)
Project Team• Project TTL - Pablo Abriani and Andrea González
• Specialist(s) - Juan Carlos Hallak
• Government counterpart 1 - Agencia Argentina de Inversiones y Comercio Internacional
• Government counterpart 2 - INTI
• Government counterpart 3 - Secretaría de Comercio
• Government counterpart 4 - Secretaría de Transformación Productiva
Budget
Item Year 1 Source Year 2 Source Year 3 Source Total
IE Team: concept note preparation 8,000 8,000
Baseline survey cost 210,000 210,000
IE Researchers: baseline survey & analysis 8,000 8,000
IE follow up surveys 125,000 125,000 250,000
IE Researchers: follow up survey & analysis 8,000 8,000
IE Coordinator 4,500 4,500
IE Dissemination workshops - 10,000 10,000
IE travel -
Total cost of impact evaluation
Cost of intervention(s) (diagnostic+TA) 690,000 690,000
Platform 200,000 200,000
Miscellanous 50,000 50,000 100,000
Thank you!
Colombia productivaTarget for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
Colombia Productiva
• The program consists of three stages: (1) a comprehensive diagnostic of the firm in 7 areas of performance, (2) 208 hours of technical assistance by specialized consultants over a 6 month period, for the areas in which the firm is found to be lacking, for a subset of the firms participating in the first stage, and (3) participation in business match-making forum, for a subset of the firms participating in the stages 1 and 2 (stage 3 subject to sample size and funding availability).
• Firms will be evaluated on the following areas of performance:• Human resource management• Energy efficiency• Adoption of Information Technologies (IT)• Processes’ optimization• Quality standards• Products’ readiness for international markets• Management practices for exporting
• The agency in charge of implementation will be Programa de Transformación Productiva (PTP), a unit of the Ministry of Trade, Industry and Tourism of Colombia. PTP will work with the agency for export promotion, ProColombia, as an ally. ProColombia will be directly in charge of implementing stage 3.
• The program aims to (1) increase firms’ productivity; and (2) increase firms’ exporting capabilities measured by exports growth and continuity.
Colombia Productiva
• The expected cost of the program is of USD 1,302,000 (an estimated cost of USD 460 per diagnostic for 200 firms plus USD 13,000 per 208 hours of consulting support for 100 firms) + USD 703,000 (an estimated cost of USD 460 per diagnostic plus USD 13,000 per 208 hours of consulting support for 50 additional firms + USD 30,000 for business match-making forum). Stage 3 subject to sample size and funding availability.
• PTP will reach 200 firms in the first wave of the program in the diagnostic stage and 100 of them with the technical assistance package. (250 in the first wave and 150 with technical assistance + 50 with participation in match-making forum, if funding and sample size permit including Stage 3).
• Participating firms will be required to pay for part of the technical assistance they will receive [a flat rate per firm].
• Eligible firms must (1) be at least 3 years old (formally registered at the chamber of commerce) (2) be able to show sales of at least 500 minimum monthly wages on the last year, (3) not have exported more than USD TBD on any of last three years; (4) be a producer and not only a trader; and (5) belong to a set of specific sectors [chosen to ensure the necessary sample size for evaluation]
• The program will be aimed to all eligible firms, regardless of their location.
Theory of Change
LONG-TERM OUTCOMES
More than 24 months
SHORT-TERM OUTCOMES
(3 to 12 months)OUTPUTS
Diagnostics of firms in 7
areas of performanceStandardized processes
ACTIVITIES
Design of diagnostic
toolProductivity
Production of
contents for each
module of technical
assistance
Energy consumption per
unit producedHours of technical
assistance
Comtinuous exportimg
activity
MEDIUM-TERM OUTCOMES
(12 to 24 months)
Quality standards
certification
ProfitabilityExports growth
Adoption of Information
Technologies (IT)
Training of personnel
who will apply
diagnostic tool
Definition of final
eligibility criteria for
participation
Training and hiring of
consultants
Invitation to
participate
Product readiness for
international markets
Exports growthBusiness match-making
forum (subject to
sample size and funding
availability for Stage 3)
Organization of business
match-making forum (subject
to sample size and funding
availability for Stage 3)
Impact Evaluation Questions
• What is the impact of offering firms careful diagnostics of their performance in different areas, on their productivity, profitability, exports growth, and continuity as exporters?
• What is the impact of offering firms not only careful diagnostics of their performance in different areas but also technical assistance during a 6 month period, on their productivity, profitability, exports growth, and and continuity as exporters?
• What is the impact of offering firms participation in a business match-making forum after offering both a careful diagnostics of their performance in different areas plus technical assistance during a 6 month period, on their productivity, profitability, exports growth, and and continuity as exporters?
Main results indicators and data
LONG-TERM OUTCOMES
More than 24 months
SHORT-TERM OUTCOMES
(3 to 12 months)
Number of standardized
processes
Sales per worker
Change in KWh per
dollar of sales
Number of consecutive
semesters with exports
of at least USD 10.000
MEDIUM-TERM OUTCOMES
(12 to 24 months)
Number of certifications
obtained
Return on Assets (ROA)Number of new software
solutions adopted
Number of products that
comply with market
requirements
Exports growth in USD
Number of days from
order to shipment
Number of new products
exported
Number of new export
destinations
Exports growth in USD
Number of new products
exported
Number of new export
destinations
Change in managerial
and organizational
practices
Data availability
LONG-TERM OUTCOMES
More than 24 months
SHORT-TERM OUTCOMES
(3 to 12 months)
Number of standardized
processes:
Survey
Sales per worker:
Financial statements /
Survey + Official records
of formal employment
(PILA)Change in KWh per
dollar of sales:
Financial statements /
Survey
Number of consecutive
semesters with exports
of at least X value:
Official trade records
(DIAN- DANE)
MEDIUM-TERM OUTCOMES
(12 to 24 months)
Number of certifications
obtained - Survey
Return on Assets (ROA):
Financial statements /
Survey
Number of new software
solutions:
Survey
Number of products that
comply with market
requirements - Survey
Exports growth in USD:
Official trade records
(DIAN - DANE)
Number of days from
order to shipment:
Survey
Number of new products
exported:
Official trade records
(DIAN - DANE)
Number of new export
destinations:
Official trade records
(DIAN - DANE)
Exports growth in USD:
Official trade records
(DIAN- DANE)
Number of new products
exported:
Official trade records
(DIAN- DANE)
Number of new export
destinations:
Official trade records
(DIAN- DANE)
Change in managerial
and organizational
practices:
Survey
Methodology
• Participants will be invited to apply to be part of the program. They will then be randomly selected into one of three groups, a control group and two treatment groups
• Modules will be added to an official survey from the National Office of Statistics to collect information that is not available from administrative records.
300 firms
Control Group
100
Application to program
Arm 2
100
Application to program +
Diagnostic of 7 areas of
performance + Technical
Assistance
Arm 1
100
Application to program +
Diagnostic of 7 areas of
performance
Methodology (alternative with Stage 3)
• Participants will be invited to apply to be part of the program. They will then be randomly selected into one of four groups, a control group and three treatment groups
• Modules will be added to an official survey from the National Office of Statistics to collect information that is not available from administrative records.
350 firms
Control Group
100
Application to program
Arm 2
100
Application to program +
Diagnostic of 7 areas of
performance + Technical
Assistance
Arm 1
100
Application to program +
Diagnostic of 7 areas of
performance
Arm 3
50
Application to program +
Diagnostic of 7 areas of
performance + Technical
Assistance + participation in
business match-making forum
Challenges
• Challenge 1: Take up• Possible reasons: poor marketing, lack of knowledge (don’t know what they don’t know). Will
require using all channels of communication to attract eligible firms.
• Challenge 2: Obtaining information from control group• Possible reasons: Lack of contact with government agency after application and rejection.
Will be addressed by including module for all firms in the evaluation in official survey. Alternatively, providing information should be made easy and awarded with something of value to the firm (i.e. comparison to firms in the US after answering?)
• Challenge 3: Contamination of control group• Possible reasons: participation in other government programs or other programs. Will be
addressed by collecting information about program participation.
• Challenge 4: Implementation failures• Possible reasons: quality of consultants and contents of technical assistance as well as
standardized delivery. Will require careful planning and training as well as standardized guidelines for assistance.
Impact Evaluation Timeline
Concept
Aug 2017
Baseline
June 2017
Implementation
June 2017-Feb 2018
Follow up 1
June 2018
Follow up 2 and 3
March and Sept 2019
• Funding application
• Approval process
• Cleaned dataset
• Data analysis• Client
workshop
• Monitoring report
• Lessons note
• Cleaned dataset
• Data analysis• Client
workshop• Technical
paper• Policy note
• Cleaned dataset
• Data analysis
Thank you!
Discussion
Sebrae na sua EmpresaTarget for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
Intervention(s) to be Evaluated
• Describe the interventions to be evaluated• Which agencies are involved? Sebrae-RJ
• What are key overall goals? Increase adoption of ‘best’ business practices (ideally via Sebrae’s material)
• What are the components? 1 hour in loco visit where consultants run a diagnostic on business practices
• Eligibility: micro (80%) and small (20%) formal firms
• Reach: Metropolitan region of Rio de Janeiro
Monday
Theory of Change
LONG-TERM OUTCOMESInterventionsOUTPUTSACTIVITIES Short-TERM OUTCOMES
Consultants visit firms to collect information on business practices and entrepreneur’s profile
# of firms visited
# of firms that accessed the materials
Emails and SMS sent to firms with links to materials (free)
# of ‘best’ business practices (BBP) adopted
# of firms that contacted Sebrae-RJ for further support
Increased sales Higher profitsHigher productivityMore employees
Diagnostic on business practices
Adoption of BBP according to entrepreneurs profile
Diagnostic on entrepreneur’s profile
Feedback with benchmark on best practices
‘Entrepreneur’s spirit’Feedback with benchmark on best practices + info on expected returns + videos with testimony of successful cases
Feedback with benchmark on best practices + info on expected returns
Standard feedback (current approach)
Adoption of BBP via Sebrae’smaterials/services
Impact Evaluation Questions
• Overall: What is the impact of best business practices on firm’s performance?
• Phase 1: 2017 (refine diagnostic + pilot)
• Diagnostic with the intention to pre-identify high growth potentials
• Pilot:• Specific question 1: does feedback (with benchmark) on business practices affect
adoption of best practices
• Specific 2: Is the adoption of best practices via Sebrae’s materials?
• Specific question 3: is adoption different according to entrepreneur’s profile/expectations?
• Specific question 4: Do reminders via SMS affect access to and use of Sebrae’s materials?
Impact Evaluation Questions• Overall: What is the impact of best business practices on firm’s performance
• Phase 2: 2018
• Specific question 1 (lack of info): does feedback (with benchmark) on business practices affect adoption of best
practices
• Specific question 2 (lack of info + misperception): does benchmark feedback + info on expected returns associated
to adoption of business practices affect adoption of best practices?
• Specific question 3 (lack of info + misperception + lack of motivation): does benchmark feedback + info on returns +
demonstration of successful cases affect adoption of best practices?
• Specific question 4: Is the adoption via Sebrae’s materials?
• Specific question 5: Do (SMS) reminders affect use of Sebrae’s materials?
• Specific question 6: Is adoption different according to entrepreneur’s profile?
• Main indicators and data
• Indicators: business practices, entrepreneur’s spirit, subjective expectations, and number of employees, wages, revenue, and profits
• Survey data collected in loco during consultants’ visits and administrative data
Methodology Total sample size
5000
Arm 2: Current Approach
+ Benchmarking
[n = 1000]
Arm 1: Current Approach
[Diagnosis + Feedback]
[n = 1000]
Arm 4: Arm 3 +
Short videos with testimony
of successful entrepreneurs
[n = 1000]
Arm 3: Arm 2 +
Information on Returns
[n = 1000]
Control Group
[1000]
[No intervention]
Arm 1:
[333]
[1 SMS]
Arm 2:
[333]
[2 SMS]
Arm 3:
[333]
[3 SMS]
Arm 1:
[333]
[1 SMS]
Arm 2:
[333]
[2 SMS]
Arm 3:
[333]
[3 SMS]
Arm 1:
[333]
[1 SMS]
Arm 2:
[333]
[2 SMS]
Arm 3:
[333]
[3 SMS]
Arm 1:
[333]
[1 SMS]
Arm 2:
[333]
[2 SMS]
Arm 3:
[333]
[3 SMS]
We may stratify the sample according to entrepreneur’s profile before randomizing to be able to test for heterogeneous effects
Challenges
• Discuss potential challenges/risks and how to tackle/mitigate them
• Take up: the risk of having firms denying to answer the survey exist but it shouldn’t affect take-up because there is a long backup list where new firms can be drawn from
• Contamination of control group: this is a real risk and we are considering a cluster randomization at neighborhood level to mitigate the problem
• Implementation failures:
• Measuring spillovers: if cluster randomized trial, we will explore the idea of conducting a saturation experiment where the proportion of treated firms varies in each neighborhood
Impact Evaluation Timeline
Concept
Aug 2017
Baseline
July-Sept 2018
Implementation
July-Sept 2018
Follow up 1
July-Sept 2019
Follow up 2
July-Sept 2020
Final dissemination
March 2021
Funding applicationApproval process
Cleaned datasetData analysisClient workshop
Monitoring reportLessons note
Cleaned datasetData analysisClient workshopTechnical paperPolicy note
Cleaned datasetData analysis
Technical paperPolicy noteClient workshop
For 2017: • Non-experimental ex-post IE to measure effects on adoption of business
practices (assessment of the current version of the program)• Pilot to measure effects of providing benchmarking based on the practices
adopted by the best peers in a given sector • Experiment the number of reminders via SMS seeking to increase adoption
of good practices
Thank you!
March 2017, Mexico
Entrepreneur versus Ecosystem?Examining the Impact of Radical Innovation
Programs on Firm Growth
Motivation
Source: BCRP, RICYT, Concytec.
Total Factor Productivity slowdown in Peru
3,0
6,3
5,2
3,7
-1
1
3
5
7
1997-2003 2004-2010 2011-2013 2014-2015
Empleo ajustado por Capital HumanoCapital FísicoProductividad Total de FactoresPBI Potencial
Growth Potential(annual change as % of GDP)
117Source: RICYT
Innovation(R&D as % of GDP)
0,15% 0,20%
0,50% 0,50%
0,80%
1,20%
2,40%
2,73%
0,00%
0,50%
1,00%
1,50%
2,00%
2,50%
3,00%
Perú Colombia Chile México Prom.
LatAm y
Caribe
Brasil OCDE EEUU
• Innovation (by SMEs) a key determinant of TFP… but also low!
Motivation
2 novel initiatives
118
Entrepreneur exposure: since 2015, 2 entrepreneurs per year
119
Gastronomic tourism
2 novel initiatives
120
Gastronomic tourism
2 novel initiatives
“Innovation Champion”
Behavior change: Future vision Collective message Revaloración de
productos nativos Sinergias entre todos
los actores del ecosistema
A success story
Scaling up 2 novel initiatives
121
2 Interventions
T2: “Ecosystem” (firm-level)
Change the behavior
Innovation Champion
Helps to connect the ecosystem
T1: “Entrepreneur-exposure”
Change the mind-set of the open to change
- Risk-Taking
- Ambition for growth
- Appetite for experimentation
- De-stigmatizing failure
Research Questions
122
1. What is the impact of “entrepreneur versusecosystem” innovation interventions on firmperformance?
Main outcomes Productivit Sales Profits
2. Which innovation focus (or both) drives more radical innovations?
3. What are the internal and external innovation-related outcomes that result from more radical innovations?
Internal innovation-related outcomes (4Ps) Processes Products People Pricing & mark-ups
External innovation-related outcomes (4Cs): Competition Customer Chain Collaborators
Intervention-directly-related outcomes Continuum of incremental to radical innovations (TBC)
123
16 sectorsidentified by
Government as
strategicallyimportant for
economic growth
5 sectors chosen by the
Government to focus for the
lending operation
2 sectors for the
RCT
Research Design: Sample
124
Field work to be used to select the sectorsSectors with potential big impact on innovation, productivity, and jobs.To be defined: we will ignore the subsistence level and the big firms. Largesample of micro (5+; rule out 0-5), small, and potentially medium-sized firms;large firms will be excluded.
Screening steps to identify the study sampleGovernment register plus door-to-door marking of the programPitch the program and only bring in interested candidates/firms (Self-reported interest)
Screening process to identify firms with higher potential to innovate (Data)Proxies: GPI score cardPractices-BPI (or a mixed of both)Other restrictions? (i.e., language for training, industries/products)
Target sample size 1500 firms only for the study
Research Design: screeninng process to build the sample
125
Unit of analysis: individual firm owner
Randomization: stratified at the sector level and randomization within sectors
Research Design: Experiment
Sample1500 firms (screened)
Sector A(N=750 screened into sample)
T1 (N=250 firms)
T2 (N=250 firms)
C (N=250 firms)
Sector B(N=750 screened into sample)
T1 (N=250 firms)
T2 (N=250 firms)
C (N=250 firms)
126
Baseline
Timeline
T1= 12 months
Follow up 1
T2= 24 months
Follow up 2
Discussion
Ethiopia NQITarget for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
Ethiopia
THE NATIONAL QUALITY INFRASTRUCTURE (NQI)
TECHNICAL INFRASTRUCTURE GOVERNANCE
NQI Technical Committee (NQITC)
MOST National Science, Technology and Innovation Council (NSTIC)
National Quality Forum (NQF)
REGULATORY GOVERNANCE
MOT Inspection and Regulatory Directorate
Ethiopian Consumer Affairs
Ministry 1
Ministry 2
Ministry 3
Ministry 4
Ministry n
MOT
WTO Notification Authority
Import and Export Goods Co-ordination Control
Office for Co-ordination of Technical Regulations (TRO)
Legal Metrology
WTO National Enquiry Point
Technical Regulations Forum
MOST
NQI Directorate
TECHNICAL INFRASTRUCTURE
1
2
3
45
6
7
8
9
1011
13
12
TECHNICAL INFRASTRUCTURETechnical Infrastructure Governance
MOSTNQI Directorate (General)
SPS TBT
Private Conformity Assessment service providers
Proposed Development Objective
to improve the delivery ofquality assurance services toenterprises in the targetedsectors.
Interventions to be Evaluated
• Background: Ethiopian firms struggle to export due to trouble meeting export requirements (high rejection rates from advanced markets)
• Four institutes (standards, metrology, accreditation, conformity assessment)
• Target firms: • Potential exporters in leather, textile and garment, agro processing industries
• ~500 firms
Impact Evaluation Questions
• Does the preferred access to QI services increase firms’ usage of QI services? [improving supply]
• Does the provision of tailored diagnostic support in addition to preferred access increase firms’ usage of QI services? [improving demand]
• Does the increased usage of QI services increase firm growth and productivity (profits, volume of exports, number of new export contracts, employment)? [impact]
• Does the increased demand and usage of QI services increase the quality of input from suppliers to QI service users? [spillover]
Methodology
Total Sample Size
Firms screened for
interest
300-400
Treatment 2
Preferred Access
And
Detailed Diagnostic
100-133
Treatment 1
Preferred Access
100-133
Control Group
100-133
Potential Exporters
500
Methodology / Data Collection
• Firm survey of potential exporters
• Mapping exercise of supply chains to understand linkages and potential spillovers
• Surveys of actors down the supply chain• Ex. Leather goods sector: Traders, farmers, intermediaries
• Potential export market analysis
Theory of Change
LONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS
Provision of
internationally
recognized QI services
Increased firm
compliance to
quality requirements
ACTIVITIES
Provision of QI
ServicesIncreased export
Increased
productivity
Firm growth
Provision of tailor-
made diagnostic
services
Preferred access to
QI Services
Reduced compliance
cost of businesses
Firms have a better
understanding of
quality improvements
Reduced QI service
timeIncreased quality
of suppliers down
the supply chain*
Its own theory of
change
MEDIUM-TERM OUTCOMES
Access to new
buyers
(domestic and
abroad)
Ability to meet
export
requirements
Concluding Remarks
Without Effective and Trusted NQI System,
• No Strong Enterprises፣
• Neither Development nor Competitiveness are possible.
Increasing competitiveness and jobs through matching grants: evidence from Kosovo
Target for Growth: Impact Evaluation Workshop
Mexico City
February 27th- March 3rd, 2017
Intervention to be Evaluated
• Implementing agency: Kosovo Investment and Enterprise Support Agency, KIESA (good buy-in, limited capacity)
• Overall goals: Increase SME exports and improve SME performance
• Components? Matching grant scheme (co-finance 50%) to provide (i) quality certification and standards, and (ii) access to business development services
• Scope: Plan is to support 150 SMEs with average size of grant being $16-17k implemented over 1-1.5 years (certification may take 1.5 years if firm is far from ready for certification)
• Eligibility: SME registered in Kosovo, 10-50 employees, 2 years operating, not in natural resources and not harming environment
• Reach: the program will cover SME at national level
Theory of Change
LONG-TERM OUTCOMESSHORT-TERM OUTCOMESOUTPUTS
Number of firms
applying to matching
grant scheme
ACTIVITIES
Encouragement to
help a subgroup of
firms apply to the
matching grant
scheme
Volume of exports per
firm in USD
Numbers of new
products exported and
new markets reached
Delivery of matching
grant – for product
quality certification
and standards
Delivery of matching
grant – for access to
BDS
Number of quality
certifications provided
Number of standards
implemented
Number of BDS
delivered (in total and
by categories)
Firm performance
(revenue, profit,
employment,
investment,
productivity)
MEDIUM-TERM OUTCOMES
Increased capacity and
know-how of firms and
training of employees
Impact Evaluation Questions
• What is the impact of providing business development services on firm exports, employment, and productivity?
• What is the impact of providing quality certification and standards along with business development services on firm exports, employment, and productivity?
• Main indicators and data• Using the results chain, identify and define key indicators: volume of exports per
firm, numbers of new products exported and new markets reached per firm, measures of firm performance (revenue, profit, employment, investment, productivity)
• Describe data availability: exporter customs data, Kosovo Statistics Agency firm annual survey, Tax administration data, Business registry data, application form data, we will need to conduct follow-up surveys (ideally 2 or 3)
Methodology
• Sampling: call for applications, among applicants eligibility criteria will define a cutoff point, among eligible applicants above cutoff point we will randomly select groups for treatment and control
• Expectation would be 1000 applicants out of which 300 would be above cutoff
• Matching grant scheme has funding to provide grants to 150 beneficiaries
• Depending on actual applications we will know how many firms will get treatment in each of bullets in Treatment 1
• Data will be collected via surveys and existing administrative data
Total sample size
300
Control Group
150
Treatment 2
May be added during
implementation
Treatment 1
150
• Matching grant used to access to
Business Development Services
• Matching grant for product conformity
• Matching grant used for both above
activities
Challenges• Challenge 1: Take up at application and eligibility phases
• At application phase • Very good communication campaign, an easy application process, telephone and online help• Encouragement design (drawing lessons on failed matching grants in Africa & fear that not enough firms will apply for 50-50 grant when
previous schemes in Kosovo were 75-25)
• At eligibility phase • Spell out clear and objective criteria (e.g., business plan must include X, Y, Z sections)• Ask applicants for proof of availability of co-financing to reduce drop out at implementation stage
• Challenge 2: Implementation failures – at disbursement phase• To easily disburse: make payment directly to vendor rather than firm paying upfront and getting reimbursed• Require progress reports from beneficiaries
• Challenge 3: Availability and quality of consultants in Kosovo• Beneficiaries will be provided with information on a network of consultants in the Balkan region
• Challenge 4: Political buy-in of randomized treatment and private sector reluctance to accept• Complete buy-in from KIESA CEO and good communication with public and private sector explaining transparency of
lottery
• Challenge 5: Contamination of control group• Keep good data on control firms and may enable us to measure additionality of matching grant scheme
• Challenge 6: Time frame may be too short to see export increases
Impact Evaluation Timeline
Concept
Aug 2017
Baseline
1st round
Q1 18
Implementation
Q2 18-Q2 19
Follow up 1
Q2 19
Follow up 2
Q2 20
Final dissemination
2021 (?)
• Funding application
• Approval process
• Cleaned dataset
• Data analysis• Client
workshop
• Monitoring report
• Lessons note
• Cleaned dataset
• Data analysis• Client
workshop• Technical
paper• Policy note
• Cleaned dataset
• Data analysis
• Technical paper
• Policy note• Client
workshop
Thank you!
Discussion
THE END