TaqeemCaseStudies Issue 1: The Case of Education for Employment (EFE), Morocco 1

TaqeemThe Youth Employment Network

Case Studies

M&E implementation at Education for Employment (EFE), Morocco

An overview of the technical support initiative delivered as part of the Taqeem program to increase the understanding and knowledge of Measurement and Evaluation in the MENA region.

• Conductingafollowupsurveyamongstprogrambeneficiairesisrecommendedto show evidence of achieving outcomes. Follow up surveys should allow suf-ficienttimeforoutcomestoappear,whichvariesdependingonthetrainingprogram.

• Whenfacingalargenumberofbeneficiariestofollow-upwithanddependingonthepurposeofthefollow-up,samplingmightbeadvisable,asopposedtotrack-ingallbeneficiaries.Thesamplingcanbedonebyrandomselectionwhichhelpsreduce bias in results.

• Whilewebbasedsurveyingisagreatwaytoreducetimeandresources,itposesnumerouschallengeswhichneedtobeaddressedwellinadvance.Qualitativeevaluationsareidealtodeepenquantitativefindingscomingoutofthesewebbased surveys.

• Having somebody fully dedicated to M&E is necessary for any organization working on youth employment and wishing to implement the full cycle of an M&E plan.

Key Messages Contents:

Background

M&E System

• • 2

• • 1

Findings

• • 5

Recommendations

• • 6

Next Steps

• • 7

TheYouthEmploymentNetwork(YEN)isaninter-agency partnership of the International LabourOrganization(ILO),theWorldBankandthe United Nations set up in 2001 to provide new solutions to the youth employment challenge.

* * * * *TheTaqeemCommunityofPracticeisaregionalsupport program aiming at improving results measurement of youth employment and entrepre-neurship dedicated non-governmental organiza-tions.YEN’sTaqeem case studies highlight the experiences of Taqeem member organizations.

EFE’s “Finding a Job is a Job” ProgramEducationforEmployment(EFE)isanon-profitorganizationheadquarteredinNewYork,withindependentaffiliatesinsixcountriesintheMiddleEastandNorthAfrica(MENA),Egypt,Jordan,Palestine,Morocco,TunisiaandYemen.TheEFE network aims at addressing the youth employment chal-lengeintheMENAregionbyofferingvarioustechnicalandvocationaltrainingopportunitiestounemployedyouth,withthe goal of reducing the gap between diverging youth quali-ficationsandemployers’humanresourcerequirements.EFE Morocco was founded in 2007 and currently runs four distincttrainingprograms,oneofwhichiscalledFindinga

Background

Issue 1, July 2013

TaqeemCaseStudies Issue 1: The Case of Education for Employment (EFE), Morocco 2

JobisaJob(FJIJ).FJIJisathreedayemployabilitytrainingtargetingunemployedyouth,oftenyouthwhomareinthetransitionfromschooltowork).FJIJteachesjobsearchskills,improvingtheCVandcoverletter,andhowtoper-forminajobinterview.Everycoursebringstogetheronetrainerand20to25participants.EFEMorocco’sgoalistotrain12500youthinFJIJoverafouryearperiodwhichstarted in 2011. EFE Morocco partners with youth centres anduniversitiesinthedeliveryoftheprojectandtrainingis delivered in six regions of Morocco.1TheFJIJtrainingtakesplaceundertheumbrellaoftheAl-Moradinitiative,awiderprojectfinancedbytheMasterCardFoundation.

EFE and TaqeemEFEjoinedtheTaqeeminitiativebecauseofadesiretoreformitsM&Esystem.Themonitoringandevalua-tion(M&E)functionatEFEfallsunderthePerformanceManagement department which is a centralized function overseenbyEFEheadquartersinNewYork.EachEFEaf-filiatehasanM&Emanagerresponsibleforallprojectsinthe country. In the past two years there had been a sud-

denincreaseofprojectsunderEFE’sportofolioandthecurrent system could not handle the resulting increase in data.Furthermore,datawasonlybeingcollectedattheoutputlevelandEFE’sstakeholdershavebeendemandingmoreevidenceoflongtermoutcomeresults.Finally,EFE’sdatacollectiontoolswereoutdatedandbeingmisused,responseratestoformsandsurveyswerelowandoftentimes responses were not being transcribed to the database for analysis.

EFE decided that it would pilot its new M&E approach us-ingtheFJIJprojectinMorocco.ForthepasttwelvemonthsandwiththesupportofM&Eexperts,EFEhasrolledoutaseries of revised data collection tools and while the reform processisnotyetcomplete,evidenceofprogramoutcomesismaterializingandfindingsarebeingusedtoimprovetheprogramdesign.Thefollowingparagraphsdetailthedatacollectiontoolsthathavebeenplanned,provideevidenceofsomeearlymonitoringresults,andfinallyaddressthechallenges faced in the form of recommendations and next steps.

Results chainTheresultschainfortheFJIJprogramisbasedontheassumptionthattheeducation,knowledgeandskillspro-vided by public education system in Morocco are many timesinsufficienttofindqualityemployment.Italsoas-sumesthatjobsareavailableforyoungpeoplethathavetherightsskills.Withthisinmind,thelongtermimpactoftheprojectistocontributetosocio-economicinclusionof underprivileged youth in Moroccan society. For socio-

economicinclusiontobeachieved,EFEwillneedtomeettwo outcomes: young people gain skills and knowledge onhowtoconductaneffectivejobsearchandthejobsearch is successful leading to decent employment. At the outputlevelandthebasisforEFE’sinterventionstrategyistheFJIJtrainingprogramfromwhich12,500youngpeople are expected to graduate. Figure 1 above gives an overviewoftheproject’sresultschain.

Figure 1: Overview of the results chain for EFE Morocco’s Finding a job is a job (FJIJ) project

M&E System

TaqeemCaseStudies Issue 1: The Case of Education for Employment (EFE), Morocco 3

Data collection instrumentsWhile the results chain helps us to map the logical framework of the projectdesign,itneedstobecom-plemented by a strategy for measur-ingwhetherobjectiveshavebeenachieved.Attachedtoeachentryin the results chain are indicators. While indicators tell us what data wewillbemeasured,thedatacol-lection instrument tells us how that datawillbemeasured.TheM&EplanforEFE’sFJIJprojectincludesfiveinstrumentswhichmeasureaseriesofindicators,asdescribedbelow in Table 1.

Table 1: Description of the data collection tools and main indicators of the M&E system for FJIJ

What? An online application form to which youth apply. It asks questionsonsocio-demographics,workstatus,educationandmotiva-tion.EFE’ssourcingteamdetermineseligibleapplicants(age,fam-ilyincome,workstatus,motivation)andcontactsthemforashortinterview.When? On an on-going basis. Takes 10 minutes to complete and con-tains around 15 questions.Management: DataisexportedtoExcelandfiltered.Acceptedcan-didates are forwarded to the training team who arranges the training and to the M&E team who registers data into the database.Implementation: Was improved on multiple occasions and works well,butfailstoincludeFJIJparticipantsfrompartnershipwithuni-versities.

What?Attendancesheet,managedbythetrainers.Importantsincegraduationisonlygivenifattending60%ofthesessions.Alsoincludescolumns to verify contact details of participants.When? SignedaftereverysessionofFJIJ.Management: AftereveryFJIJ,trainerssendascanoftheattendancesheettothetrainingteam,whomanuallyupdatesparticipant’sdatainthe database.Implementation: Successfully tested and implemented

What?Anonlinequestionnairesenttograduateswiththeobjectiveofmeasuringparticipants’satisfactionwiththetraining,evaluatetheinitialskillsandknowledgegainedandunderstandparticipants’shortterm plans for looking for work.When? Upon completion of the training. Takes 10-12 minutes to com-plete and contains around 20 questions.Management:DataisexportedtoExcel,filteredandincorporatedintothedatabase,thenregularlyanalysedandsharedwithprogramstaff.Implementation: One of the challenges has been a low response rate asmanyyouthdonotreceivetheemail,donothaveinternetconnec-tionorsimplyignorethecommunication.Remediessuchasincentives,reminders and face-to-face interviews are being explored.

Instrument Status Indicators Description

•No.ofapplicantstoFJIJ•Jobsearchknowl-edge at baseline•Jobsearchskillsat baseline•Attitudetowardsemployment at baseline•Motivationtogetajobatbaseline

Pilotphase(Proceduresarebe-ingrevised)

Application form and interview

•No./%ofyouthparticipants•No.oftrainingsdelivered

Roll outAttendancesheet

•%ofgraduatessatisfiedwithtrain-ers and training•No./%ofgradu-ates,self-percep-tionofeffectiveskills formed

Roll outEnd of training questionnaire

TaqeemCaseStudies Issue 1: The Case of Education for Employment (EFE), Morocco 4

What? Onlinefollowupsurveyssenttograduateswiththeobjectiveoffollowinggraduates’jobsearchactivitiesandoutcomes.Addressestopicssuchasuseofdifferentjobsearchtechniques,timeinvested,applicationssent,interviewsgranted,assessmentofFJIJcontribution,etc.When? Twoandfourmonthsafterthetrainingended.Takes10-12minutes to complete and contains around 20 questions.Management: DataisexportedtoExcel,filteredandincorporatedintothedatabase,thenregularlyanalysedandsharedwithprogramstaff.Implementation: Never fully implemented due to the time burden onstaffandlowresponserates.Thepilotsurveyhasbeentestedwith200graduatesandreceivedfeedbackfromthebeneficiaries.Ithasbeen recommended that EFE drops the 2 month follow up survey to reduce the burden of answer for graduates as well as focus on the intermediatejoboutcomes(morerelevanttomeasureafter4months).

What? GroupdiscussionsmoderatedbyEFEMorocco’sM&Emanag-er,bringingtogether8-10FJIJgraduates,involving4-6keyquestionsusedtocomplementquantitativefindingsfromtheonlinesurveys.Themoderatorisaccompaniedbyanotetakerwhorecordsallconver-sation.When? Held2-3timesperyear,eachlastingaround60-120minutes.Management:Toanalysethedata,EFEwillusea“ParticipatoryLive-lihoods System Analysis” 2,anefficienttacticforinterpretingresults.Implementation: To be performed

•AssessmentofFJIJcontributionfor many indicators

PilotphaseFocus groups

•No.ofgraduateswhoused/benefit-edfromtheskills/knowledge formed•No.ofgraduatesgettingadmittedtoajob•Averagenumberofjobsappliedtoin the last months•No.ofinterviewsgranted in the last months

PilotphaseTwo and four month follow-up surveys

Figure2showsEFEMorocco’sdatacollectioncyclestartingwiththeapplicationformandendingwiththe4monthfol-lowupsurvey.Afullcycleiscompletedinfivemonths.

Figure 2: Overview of the data collection workflow of the M&E system for FJIJ

Data managementEFEMorocco’sM&EdatabaseisaMicrosoftExcelspreadsheet where all administrative and survey dataisinput,cleanedandanalysed.Whilenotanidealtool,itwasdecidedtouseexcelforthisop-erationtoprovideasimple,low-costsolutionandonethatstaffalreadyhadtheskillstooperate.

Data from the online surveys can be automati-cally exported to the spreadsheet while data from theattendancesheetsistranscribedfrompaper.TheM&Emanagerisresponsibleforcontrollingthe quality of the data which includes the arduous task of matching records from the various surveys to the same individual.

EFE Morocco is seeking to migrate to a more powerfuldatabasesoftwareinordertobetterdealwith the growing number of data entries and to gainflexibilityintheanalysisperformed.

TaqeemCaseStudies Issue 1: The Case of Education for Employment (EFE), Morocco 5

TheaimofthissectionistoshowcaseEFEMorocco’sincomingdataresultsanddemonstratehowitcanbedisplayedandshared.ThedataprovidesevidencethatEFEisonitswaytoachievingitsobjectivesforFJIJ.ResultspresentedbelowlookatthreeofEFE’smainindicators,namelyjobsearchconfidence,jobsearchtechniquesandemployment.Theseresultsshouldbeinterpretedwithprecaution,mainlyfortworeasons

• Responseratesamonggraduatesremainlow.Itisunknownyetiftherespondentssignificantlydifferfromthenon-respondents,andifresultsobtainedfromthesamplecanbeextendedtothewholepopulationofFJIJgraduates.

• Assumingthatthecollecteddatacouldbeextendedtothewholepopulationofgraduates,theproblemofcausal-itywouldpersist.Causalitybetweentheprogramandtheoutcomescanonlybeassumed,sincenocounterfactualgroup exists.

Self-perception of effective skills formedEFEasksgraduatestoassesstheirlevelofconfidencebeforeandaftertheFJIJtrainingregardingmanyaspectsofjobsearchtechniquesandskills.ResultshighlightedinFigure3showthatintheseventopicscoveredduringthetraining,thegraduates’levelofconfidenceuponthecompletionofthetrainingismuchhigherthanitwasbeforethetraining.

Use of acquired skillsEFEMoroccoisalsointerestedinknowingiftechniquesgraduatesusetofindajobafterthetraininghavechangedcomparedtobefore.Figure4presentsthefrequencyofuseofsixdifferentjobsearchtechniques.Itsuggeststhatthefrequencyofuseincreasesafterthetraining,comparedto

before.ItcanbenoticedthatanessentialcomponentoftheFJIJtraining,usingyourpersonalcontactsforjobsearch(networking),onlyshownslightimprovement.Ifjudgedrelevant,EFEMoroccocouldthereforereinforcethisportionof the training.

Figure 3: “Before and After” comparison of the level of confidence in performing job search related activities, for 202 FJIJ graduates (N=182)

Figure 4: “Before and After” of the use of job search techniques, for 2012 graduates who looked for a job both before and after (N=24)

Findings

TaqeemCaseStudies Issue 1: The Case of Education for Employment (EFE), Morocco 6

Admission to a jobFigure5illustratesresultsofFJIJgraduatesgettingadmittedtoajob2and4monthsaftertraining.Thefindingstelluslessthanhalfofthegraduatesfoundajob.Interestingly,graduateswhofoundajobdon’tseemtoattributethissuccesstotheFJIJtrainingalthoughwedonothaveinformationtotelluswhattheyattributetheirsuccessto.Bringingtogethersuc-cessfuljobseekersinafocusgrouptobetterunderstandthisdatawouldbeadvisable.

 

47%  53%  

2  months  a*er  FJIJ  

Admi+ed  to  a  job  

S3ll  looking  for  a  job  

37%  63%  

4  months  a*er  FJIJ   The  successful  job  seekers  (47%  and  37%)  were  asked  if  FJIJ  had  contributed  to  this  

positive  outcome  Average  answer  located  

between  “little  helpful”  and  “somewhat  helpful”  

TheseselectedfindingsareexamplesofgoodguidanceprovidedbytheM&Esystemtoimprovetheinternalmanage-mentoftheproject(programdelivery,restructurationofthecurriculum,etc.).SinceEFEMoroccocollectsallinforma-tionattheindividuallevel,furtherdisaggregationanalysisbysocio-demographicvariablescanbeperformedontheseresults,andwouldbehelpdetermineifresultsapplytoallcategoriesofgraduates.

Based on the challenges faced by EFE Morocco in the implementationoftheirM&Esystem,severalrecommen-dationscanbeformulated.Thecurrentcasestudyfocuseson four main recommendations.

Survey random sample of your beneficia-riesEFE regularly sends online surveys to hundreds of youth per month. All youth who participate in EFE trainings are sent both an end of training survey and two follow up surveys.Responseratestothesurveyshaveoftenbeenlow;inarecentEFEsurvey,only24%ofsurveyedyouthprovided a response.

Inthefuture,EFEdoesnotneedtosendtheirsurveystoall training participants and instead can only send the surveytoasmallersampleofrespondents,creatingasample frame.

Calculatingasampleframecanbedifficultasonewouldneedstoknowthepopulationsize,theconfidenceinter-valandtheconfidencelevel.ForamediumsizedNGOlikeEFE,acoupleofintuitivetipscanhelptheM&Eteamimprove validity:

• Only survey a sample of the training participants: as arule,thebiggerthesampleframe,themoreac-curate the results will be. But this depends on a few

conditions.Inthefuture,EFEshouldonlyincludeapercentage of the training participants in the sam-pling frame. Determining the sampling frame through random selection would help reduce bias.

• Reduce non-responses: each non-response increases thelikelihoodforbias.Thismeansthatpeoplewhodonotrespondhavethepotentialtobedifferentfromthosewhodorespond,reducingtherepresentative-nessofthesample.Asaruleofthumb,thenon-re-sponserateshouldbenomorethan20%.

• Increasing response rate: strategies for increasing responserateincludeofferinganincentive,sendingseveral reminders and personalizing communication.

The M&E databaseEFEMoroccohasreliedonMicrosoftExceltocompileandanalysedataaboutFJIJ.WhileExcelisgoodforonetimeanalysis,asnewcolumnsandrowsgetadded,formulasandrangesneedtobemodified.Furthermore,workingwith an external online survey platform means regularly exporting data and integrating it to an Excel database. Thisrequiressettingupcomplexformulasthatarehardtounderstand and adapt.

EFEMorocco,aswellaspractitionersexpectingtofacethesameissues,shouldseekadatamanagementandanalysissoftwaretoallowthemtobetterprocessthelargeamountsanddifferenttypesofdatathatarebeing

Figure 5: Percentage of job seeking FJIJ graduates who found a job, for 2012 FJIJ graduates (N=25, N=51)

Recommendations

TaqeemCaseStudies Issue 1: The Case of Education for Employment (EFE), Morocco 7

collected. A more harmonized approach which provides integrated surveying and database functions would be beneficial.A“cloud”basedsystemwouldmeanthedatawould be available anywhere where there is an internet connectionandcouldeasilybesharedwithstaffandstakeholders.

Using a web based survey instrumentEFE has been using the popular platform SurveyMonkey for the past two years though their success using this tool isnotyetproven.EFE’srelianceonawebbasedsurveyinstrumentposesanumberofchallenges,includinglowInternetconnectivity,useof“open”surveys(anyonewiththesurvey’sURLcansubmitanentry)andcreationof“surveyfatigue”amongbeneficiaries.

Some recommendations are provided to help EFE increase thequalityofSurveyMonkeydata.TheyareusingSur-veyMonkeytoembedrespondents’nameswhensendingoutthesurveys.Thiswillhopefullyboostertheresponserateandcontributetoerradicate“open”surveys.Theyarealsointheprocessofdeletingthefollowupaftertwomonthonlytokeeptheoneafterfourmonths,hopingtoreduce“surveyfatigue”.Offeringincentivesforansweringisalsobeingdiscussed,aswellasfindingways:

• EFEhasbeenusing“open”surveysmeaninganyonewiththesurvey’sURLcansubmitanentry.Inthefuture,EFEcanrestrictaccesstothesurveysonlytothose email addresses that have been approved.

• Youngpeoplemostlikelyhave“surveyfatigue”mean-ingtheyoftenreceiverequestsforinformationandfeedbackoveremail.EFEshouldfindawaytomaketheir request unique such as providing a creative sloganinthesubjectlineorpersonallyaddressingtherespondent in each request.

• Increasetheincentivesofferedtorespondents

Sustainability of the M&E systemEFEMoroccofaceshighstaffturnover,especiallyintheM&Efunction.Thishasmeantsignificantdelaysinimple-menting the new M&E system. Considerable time delays wereobservedintheimplementationoftheM&Etools,aswell as a lacking ability to manage and use the collected data.Practitionersanticipatingasimilarchallengeshouldensure that the M&E component becomes sustainable by being integrated to all of Units in the organization.

Piloting focus groupsOrganizingfocusgroupsamongstFJIJgraduateshasbeendifficult.Focusgroupstendtobelabourandtimeinten-siveforstaff,asgraduatesareoftenunavailabledueothercommitments like employment. Graduates are also some-timesscepticalastowhytheyarebeingcontacted,eventhough careful precautions are taken regarding the word-ingoftheinvitationmessage(emailandtelephone).Asaresultofthesechallenges,theEFE’sfirstpilotfocusgrouphad to be cancelled because no participants showed up and a second one was conducted with only four of eight confirmedparticipantsinattendance.Inthefuture,EFEplanstotrytoovercomethisissuebyofferingalargerin-centivetoyouth.PlanningisessentialandEFEwillhaveto send meeting multiple reminders to participants.

Revisit M&E system for second major out-comeAsecondmajorobjective(intendedoutcome)oftheFJIJprogram is to build the capacity of partner institutions to betterserveyouth,usingexamplesfromsuccessfulEFEprograms. EFE is currently developing the results chain and indicators tied to this outcome which will be followed by the design of data collection.

On-going technical assistanceTheSwissAcademyforDevelopment,aconsultancyfirmspecializedinmonitoringandevaluation,hasprovidedexternal expertise to EFE in this reform process. Upon

Next Steps

TaqeemCaseStudies Issue 1: The Case of Education for Employment (EFE), Morocco 8

validationandfullrolloutofthesysteminMorocco,EFEislookingforwardtoreplicatethemodeltoEFEaffiliatesinTunisia,EgyptandYemen.

Pipeline impact evaluationFinally,whenafunctioningandstrongM&Esystemisinplace,andifjudgedrelevantaccordingtofutureplansofFJIJ,animpactevaluation(IE)couldbeperformedonthetrainingprogram.AstudytoassessthefeasibilityofconductinganIEofFJIJwasconductedbySAD.Itwasrecommendedtopostponetheevaluationuntiltargetingandadmissionprocedureswererevised.Itwasalsoconsideredriskyfromacost-benefitperspectivetoevaluateaninterventionwhich

Notes:

1 Partnershipswereoriginallyplannedtobeestablishedwith30youthcentresand3universities.

2 Aqualitativeandsystematicapproachtoresult-orientedM&E,initiallydevelopedtoplanandmeasureoutcomesandimpactsofcommunitybasedlivelihoodinterventions,adapted and combined with FGDs to reduce the workload for recording and analyzing while still pursuing a systematic approach.

ducitharumsunt,niendiossimposadmintiasimresideoditatisesdolorionsequaepla-borematemipsundiapresipitrevitadempeditiostiaestorest,samdoloresendaepudae