View
216
Download
2
Category
Preview:
Citation preview
1
Impact Assessment Development and overview
Paul Foley, Francesco Mureddu and Alberto Savodelli
pdfoley@tech4i2.com
JRC-IPTS MIREIAExperts' Workshop on Measuring the Impact of
eInclusion Intermediaries in EuropeGranada, 29th May 2013
The views expressed by the authors are not necessarily those of the EC
Thanks to all the intermediary personnel, survey participants and advisers that have taken part in the
project.
211.45
Context, Conceptualisation, & approach
EU27 Unemployment rate 12.1% March 2013, 11% March 2012March unemployment – ES 27%, IE 14% IT 12%, PL 11%,
Youth unemployment (under 25) EU27 in 2012 22.8% 2012 youth unemp – ES 53.2%, IE 30.4% IT 35.3%, PL 26.5%,
DAE Actions – 57; digital literacy, 59; digital skills for jobs, 60; women and ICT, 63; accessibility
Dublin #da13skills #da13smart
311.45
Context
5
Cap Gemini (2012) Benchlearning Report:-• eInlcusion intermediaries have difficulties in defining targets
and planning monitoring or evaluation activities. • Where evaluation was undertaken activities were generally
regarded as immature. • Projects and policies ‘speak different languages’.Digital Britain report (2009) - Lack of appropriate alignment of
the initiatives to ensure that they are efficiently delivered and complement each other.
The MIREIA Impact Assessment Tool should help to overcome this lack of understanding.
Conceptualisation
6
Government• Achieve policy goals• As cost efficiently as possible• ‘What works?’• Share better practices
Participants• Enter a course that will improve life chances• Share experiences to enhance impact for others
Intermediaries• Achieve organisation goals• As cost efficiently as possible• As effectively as possible
MIREIA Impact Assessment Framework
Conceptualisation
7
Methodology underpinned by theoretical considerations• Institutional theory - Role of intermediaries• New Institutionalism - Cultural explanations for behaviour• Theories of change - Interventions and outcomes• Organisational learning - Feedback into the planning process
Wide variety of theoretical perspectives that can be utilised toexplain the multi-dimensional construct of eInclusion and
the role of intermediaries.
Conceptualisation
8
Misuraca G., Torrecillas, C., (2012), 'Methodological approach for developing the MIREIA eInclusion Intermediaries
Approach
10
Participants1.Online questionnaire for
current cohort
2.Online questionnaire for
previous cohort
3.Online questionnaire for
counter-factual group
Intermediaries1.General information about the intermediary
2.Intermediary inclusion activities/initiatives
3.Relative intensity of initiatives
4.Evaluation criteria previously used
5.Relative importance of evaluation criteria
6.Importance of evaluation criteria
7.Performance assessment model
8.Instruments feedback questionnaire
Eleven instruments or tools underpin the approach
Counterfactual considerations
1112.05
Evaluating the impact of policies is a complex task because one would like to know what would have been the value for a given output/outcome variable in the absence of the incentive. But evaluators cannot know what would have been the behaviour of a treated subject in the absence of treatment. Then the main issue is to identify the appropriate control group in order to estimate the counterfactual
Obstacles in estimating the counterfactual•The 'selection bias’ consists in the fact that target population differs from counterfactual population due to pre-intervention features. A solution is the introduction of identification hypothesis stating that pre-intervention variables are sufficient to 'reconstruct' the control group of non-beneficiaries (counterfactual)
•The presence of spontaneous dynamics, due to the fact that target population differs from control population for the trend of the result variable. A solution is the introduction of an identification hypothesis to take in consideration the spontaneous dynamics of the result variable trend
Propensity Score Matching (Instruments 9)For each initiative that has been treated, it is possible to find at least one non-treated
initiative that is “close” enough to the treated counterpart, meaning that it exhibits a value for the propensity score very similar to the one observed for the treated initiative. The propensity score is defined as the conditional probability of receiving the treatment. After having computed the propensity scores for all the firms in the dataset, it is possible to use this value to match firms in the treated group with at least one firm in the control group. This approach is useful for ex-post evaluation of impacts as in the case of Guadalinfo and INNECIA initiatives (Spanish case) that were concluded before MIREAIA project activities have started.
Randomised controlled trials (Instruments 9)When the treatment is made available to firms/organizations on the basis of a random
process, we do not expect structural differences between those who are treated (and receive funds or support) and those who are not, so that we can use the non-supported organizations as a control group for comparison with the former group. This approach is useful for impact evaluation of planned activities as in the case of Pane_Internet_Lavoro initiative (Italian case) that has been both planned and evaluated during MIREAIA project life-cycle.
Of
12
Counterfactual impact evaluation and instruments 7a and 7b
Propensity Score Matching Pathway
The data gathering campaign is still open
59 valid set of responses
280 valid set of responses
Counterfactual Impact Evaluation
for MIREIA IAF
Counterfactual Impact Evaluation
for MIREIA IAF
Assignment to the treatment based on project/entrepreneur
characteristics
yes
no
Propensity Score Matching Procedure
14
By using the propensity score matching procedure we are able to define a target (treated) and a control (non-treated) group so as to assess the impact of the treatment on the relative performance of the subjects avoiding selection and spontaneous dynamics biases
Example: Guadalinfo-INNECIA (Spanish case)
Impact variables• Number of employees, number of customers, stakeholders' commitment• Survival of the initiative, success of the initiative, financial stability• ICT knowledge of the responsible, entrepreneurial knowledge of the responsible
Matching variables• Age, level of education, gender• Former involvement in social innovation activities, support from other initiatives• Work experience, previous employment sector, type of product
Propensity Score Matching Results
Headline results from the Spanish Guadalinfo case study
15
Variables
Matching MethodologiesKernel
matchingRadius
matchingStratification
matching
Number of employees + + +
Number of customers + + +
Survival of the initiative + + +
Success of the initiative + + +
Financial stability + + +
Stakeholders' commitment + + +
ICT knowledge of the responsible + + +Entrepreneurial knowledge of the responsible + + +
Randomised Control Trial pathway
The data gathering campaign is still open
99 valid set of responses
21 valid set of responses
61 valid set of responses
Impacts evaluation (outside MIREIA
initiative)
Impacts evaluation (outside MIREIA
initiative)
Outcomes evaluation (outside MIREIA initiative)
Outcomes evaluation (outside MIREIA initiative)
Open call; priority list based on subscription order
yes
no
Randomised Control Trial ProcedureBy using Randomised Control Trial procedure we are able to randomly define a target (treated) and a control (non-treated) group so as to assess the impact of the treatment on the relative performance based on two sample of the same target population
Example: Pane_Internet_Lavoro (Italian Case)Impact variables• Behavioral changes in using internet for job searching (e.g. h/week of internet usage for job searching; monthly number of job searching on line advertisement replied; monthly number of job searching portals consulted; monthly numbers of job offer received by e-mail) • Behavioral changes in job-searching methods (e.g. direct contacts; relatives/friends contacts; Internet social network contacts; company on-line forms; job intermediaries on line forms)Control variables• Age, level of education, gender• Labor conditions in the last three years: sector; average yearly gross salary; average number of working months• presence of computer and interned at home• degree of knowledge in using computer and internet
17
Randomised Control Trial ResultsHeadline results from the Pane_Internet_Lavoro Italian case study
18
Ab
solu
te n
um
ber
s
Key Behavioral changes
Proxi of expected impact
T-sud0,05
T-sud0,01
T-sud<0,001
T-sud<0,01
EUROPEAN ADDEDD VALUE
SOCIAL OUTCOMES
Attribution issues and/or time lag
OUTCOMES ADDITIONALLY
Adequacy of funding strategy to the
eInclusion needs
Efficiency in allocation and deployment of
funding
Adequacy of Risk
management approach
Private and public funding
GLOBAL IMPACT: SMART & SUSTSINABLE
GROWTH
IMPUT FOR eINCLUSION
STRATEGY CHANGES
Motivation of the Instruments developed for the study
ParticipantImpact
EvaluationTool
Performance Assessment
Tool
Evaluation of eInclusion Intermediaries’ actions
impacts per single intermediary or groups
INPUTS
PROCESSES
OUTPUTS
BEHAVIOURAL ADDITIONALLY (Process-based)
OUTCOMES ADDITIONALLY
(Resources-based)
OUTCOMES ADDITIONALLY
(Resources-based)
Aggregate eInclusion measures of
Intermediaries’ actions impacts per single
intermediary or groups
Impact Measurement
Tool
Intermediaries in various organizational forms: single organization; network; value chain
12.10 to 12.30
MIREIA methodology tested and testing instruments
ParticipantImpact
Evaluation Tool
ParticipantImpact
Evaluation Tool
Context Analysis framework
Context Analysis framework
IntermediaryPerformence
Assessment Tool
IntermediaryPerformence
Assessment Tool
Impact Measurement
Tool
Impact Measurement
Tool
Data gathering
tools
Instruments 1;2;3
Instruments 9a, 9b, 9c
Instruments 4;5;6
Instrument7
12.10 to 12.30
APPROACH
1- Analytic Hierarchy Model(Saaty, 1987; Mansar, 2006; Kahraman, Demirel, & Demirel, 2007; Saaty, 2008; Parra-Lopez, Groot, Carmona-Torres & Rossing, 2009; Saaty, & Vargas, 2012)
2 – Literature review(some example of reference used: ICT4ALL, 2008; Codagnone, 2009; Bentivegna S., Guerrieri P., 2010; Garrido, et All. 2012; Tinholt et All, 2012)
3 – Stakeholders engagement•Workshops (IRL; ITA; POL; SPA)•on-line and off-line interviews•Check-lists (instruments 4, 5, 6)
Instruments developed for the study
Impact measurement and instruments 4, 5
Instruments 4 and 5 – impact measurement criteria identification to measure an e-inclusion for employability initiative
EfficiencyEffectivenessSocial Value generatedSustainabilityAlignment (strategic; organizational; technological)Compliancy
Degree of usage (Instr. 4; Likert scale)Degree of importance (Instr. 5; Pairwise comparison)
Workshops (IRL, ITA; POL; SPA)
Interviews (on-line; off-line)
Weighting systems for the Impact measurement tool
From literature review –first set of criteria identified:
Pairwise comparison method
1. Each participant to the consultation express is pairwise opinion on criteria
2. The eigenvector of the matrix define the weighing system of each stakeholder
3. The eigenvectors’mean define the weighting system amongst participants to the consultation
ADVANTAGES:•More accurate participants responses•Transparent negotiation technique•Easier to reach consensus amongst stakeholders
Wide literature on the matter, e.g. Saaty,1997; Munda, 2004; Gasparotos et al., 2008; Nordstrom et al., 2012
Weighting systems
Measurement criteria weighting system (Italian Stakeholders feedback)
Measurement criteria weighting system (Spanish
Stakeholders feedback)
Impact measurement and instruments 6Instruments 6 – impact measurement indicators identification to measure an e-inclusion for employability initiative
Example: assessment variable for the “efficiency” criterion
3 dimensions analysed with a Likert scale:
• Degree of importance• Degree of measurability• Degree of usage
Workshops (IRL, ITA; POL; SPA)
Interviews (on-line; off-line)
Most important Impact measurement indicators on
an e-inclusion for employability action
Performance assessment and instrument 8
27
The PA model is based upon five critical success factors driving Intermediary organizations in their services provisioning (Lettieri, Borga, Masella & Savoldelli, 2004; Savoldelli, Borga & Lettieri, 2004; Codagnone, 2009; Limassol report, 2009; Osimo, De Luca & Codagnone, 2010).
Performance assessment and instrument 8
28
1 dimension analysed with a Likert scale:
• Degree of importance
Workshops (IRL, ITA; POL; SPA)
Interviews (on-line; off-line)
Most important Performance assessment
criteria to measure Intermediary’s
organization functioning
Participant instruments
2912.30
Current Cohort CounterfactualPrevious cohort
Background/Before: Gender, age, education, previous activity, ICT skills, ICT useActivity/during: Employability, basic digital skills, advanced digital skills, numeracy/literacy, start-up, community, inclusion
Activity/after: Employment/activity, training basic & advanced digital skills, numeracy/literacy, start-up, community, inclusion
Now, after/future perceptions: Employment activity & prospects, ICT use, Internet saving, income/benefits, inclusion, three years time
Participant results: Pane IT
35
12.45
Headline results from the Italian Pane_Internet_Lavoro case study
2: data gathering campaign still ongoing(*): on line interviews done three weeks after the treatment(**): sttaistical significance(***): statistical significance at p=0,05(****): statistical significance at p=0,01
Participant results: Pane IT
36
12.45
(*): on line interviews done three weeks after the treatment(**): sttaistical significance(***): statistical significance at p=0,1
Headline results from the Spanish Guadalinfo case study
37
12.50
Participant results: Guadalinfo, ES
43
Counterfactal group:-• 50% of those in paid employment received further training• Average paid employment income EUR 21,700 • 83% of those in education/training had undertaken courses
to improve their employability • 43% of those unemployed had undertaken further training
The previous cohort:-• 25% of those in paid employment have undertaken further
training • Average paid employment income EUR 23,500• 10% of those unemployed had undertaken further training
44
Indicative draft financial impact (not for quotation):-• FIT treatment removed 50% of participants (913) from unemployment -
welfare payments avg. EUR 13,000/year. FIT induced Treasury savings of EUR 11.9 million for one year.
• FIT additional unemployment impact (vs. counterfactual) was 13% of participants (237); (a) saving EUR 3.1 million
• FIT induced 25% (456) of participants into employment (self & paid) at an average income of EUR 23,500. Total income EUR 10.7m
• FIT additional employment impact (vs. counterfactual) was 10% ofparticipants (183). Total income EUR 4.3m for one year
• With a tax contribution of 10% - (b) Treasury income is EUR 0.43m• Net additional impact of FIT is EUR 3.53 million for one year (a + b).• FIT costs EUR 2.43 million per annum to run. Net benefit EUR 1.1m
Conclusions
4613.58
• The MIREIA Methodology works, but could be simpler.• Policymakers obtain an insight to the performance and
outputs from intermediaries and the extent to which these are aligned with policy objectives.
• Intermediaries obtain an insight to performance and outputs achieved, subsequent outcomes and financial impact.
• Participants can obtain a better understanding of potential benefits – employment prospects, income.
• XQuestionnaire
47
Impact Assessment Development and overview
Paul Foley, Francesco Mureddu and Alberto Savodelli
Pdfoley@tech4i2.com@pdfoley
JRC-IPTS MIREIAExperts' Workshop on Measuring the Impact of
eInclusion Intermediaries in EuropeGranada, 29th May 2013
The views expressed by the authors are not necessarily those of the EC
Recommended