AfghanistanPublicPolicyResearchOrganization

August2017TrainingManual

MonitoringandEvaluation:AHandbookAHandbook

www.appro.org.af 2

Thispageisintentionallyleftblank

www.appro.org.af 3

AcknowledgementsThismanualisbasedlargelyontheAustrianDevelopmentAgency’s“GuidelinesforProjectandProgrammeEvaluations”(2009),withadditionalmaterialfromavarietyofothersources,dulyreferencedinthefootnotes.Thefundingforthepreparationofthismanualwasprovided,inpart,bytheRoyalNetherlandsEmbassy–AfghanistanthroughagrantfortheAfghanistanRightsMonitor(ARM)project,awardedtoAPPRO-EuropeinNovember2015.AbouttheAuthorsThishandbookwascompiledbyAPPRO’sTrainingDepartmentcomprising:RozbihMihran,FauziaRahimiJamal,andSearSadat.AboutAPPROAfghanistanPublicPolicyResearchOrganization(APPRO)isanindependentsocialresearchorganizationwithamandatetopromotesocialandpolicylearningtobenefitdevelopmentandreconstructioneffortsinAfghanistanandotherlessdevelopedcountriesthroughconductingsocialscientificresearch,monitoringandevaluation,andtrainingandmentoring.APPROisregisteredwiththeMinistryofEconomyinAfghanistanasanon-profit,non-governmentorganizationandheadquarteredinKabul,AfghanistanwithofficesinMazar-eSharif(north),Herat(west),Kandahar(south),Jalalabad(east),andBamyan(center).Formoreinformation,see:www.appro.org.afAPPROisthefoundingmemberofAPPRO-Europe,anetworkassociationfordisseminatingappliedresearchfindingsfromconflictenvironments,conductingtraining,andcarryingoutevaluations.Formoreinformation,see:www.appro-europe.netAPPROalsoactsastheSecretariatfortheNationalAdvocacyCommitteeforPublicPolicy,comprisingsub-committeesonEducationFoodSecurity,AccesstoJustice,Anti-corruption,NAP1325andAccesstoHealth.Formoreinformation,see:www.nac-pp.netContact:mail@appro.org.afandmail@appro-europe.netAPPROtakesfullresponsibilityforallomissionsanderrors.©2017.AfghanistanPublicPolicyResearchOrganization.Somerightsreserved.Thispublicationmaybereproduced,storedinaretrievalsystemortransmittedfornon-commercialpurposesonlyandwithwrittencredittoAPPRO.Wherethispublicationisreproduced,storedortransmittedelectronically,alinktoAPPRO’swebsiteatwww.appro.org.afshouldbeprovided.Anyotheruseofthispublicationrequirespriorwrittenpermission,whichmaybeobtainedbywritingto:mail@appro.org.af

www.appro.org.af 4

TableofContents

Introduction ........................................................................................................................... 5Definitions ....................................................................................................................................... 5

PurposeofMonitoringandEvaluation.................................................................................... 7

DifferencesBetweenMonitoringandEvaluation .................................................................... 7

Monitoring ............................................................................................................................. 8InputMonitoring ............................................................................................................................. 8ProcessMonitoring.......................................................................................................................... 8OutputMonitoring .......................................................................................................................... 8OutcomeMonitoring ....................................................................................................................... 8ImpactMonitoring........................................................................................................................... 8ImplementationMonitoring ............................................................................................................ 8Results-basedMonitoring ................................................................................................................ 9

IndicatorsforMonitoring........................................................................................................ 9

UseofIndicatorsinPolicyMaking ........................................................................................ 11

Evaluations........................................................................................................................... 12EvaluationEthics............................................................................................................................ 12

ObjectivityofEvaluators ................................................................................................................... 12IndependenceofEvaluators ............................................................................................................. 12Participation...................................................................................................................................... 12TransparencyAndFocus ................................................................................................................... 12Reliability .......................................................................................................................................... 12CompletenessAndClarity ................................................................................................................. 12FairnessAndProtectionofPartiesInvolved ..................................................................................... 13Utility................................................................................................................................................. 13EvaluationCosts................................................................................................................................ 13

EvaluationCriteria ......................................................................................................................... 13

www.appro.org.af 5

Introduction

Thismanualispreparedasabrief,practicalguidetomonitoringandevaluationandshouldbeusedintraininginconjunctionwithAustrianDevelopmentAgency’s“GuidelinesforProjectandProgrammeEvaluations”(2009),availablefrom:https://www.oecd.org/development/evaluation/dcdndep/47069197.pdfandAPPRO’s“PolicyandInstitutionalAnalysis–AHandbook”,availablefrom:http://appro.org.af/publication/policy-and-institutional-analysis-a-handbook/

Definitions

Monitoring:Thecollectionandanalysisofsystematicdataforspecifiedindicatorsaboutaprojectorprogramwhiletheproject/programisongoing.Analysisofmonitoringdatacanprovidetheimplementersoftheproject/programandthemainstakeholderswithinformationontheextentofprogress,achievementofobjectives,andprogressintheuseofallocatedfunds.Findingsfrommonitoringareusedtoestablishtrendstowardpredefinedproject/programobjectivesandtakecorrectiveactionasnecessary.Evaluation:Theretrospectiveandsystematicassessmentofaproject,program,orselectedcomponentsofaprojectoraprogramtoestablishtherelevance,efficiency,effectiveness,impact,andsustainabilityofaproject/program/component(s)part-waythroughorandattheendoftheproject/program.Findingsfromevaluationsareusedtoforlearningandtoinformthedesignandimplementationoffuturephasesofaproject/program/component(s)orsimilarendeavorsinthefuture.Monitoringandevaluationsmaybeconductedinternallybydedicatedproject/programteammembersorexternallybyindependentassessors.1Ineithercase,monitoringandevaluationsarecarriedoutobjectivelyandbasedonempiricalevidence.LogicalFramework:Aprojectmanagementtooltocompileallcrucialinformationondesign,implementation,monitoring,andevaluationofaproject/programbydetailingandlinkingresourcesneededtodesignandimplementtheproject/program,activitiestobeundertakenmeetoverallandspecificobjectivesoftheproject/program,risksandassumptionsassociatedwiththeproject/program,andhowtheultimatesuccessorfailureoftheproject/programcanbemeasuredandverified.BaselineAssessment:Abaselineassessmentistypicallybasedonresearchintoasituation,oratargetarea,atwhichprogramminginterventionisaimed.Thebaselinereportservesasthedocumentedstartingpointofreferenceforassessingchangesandimpactsbroughtthroughinterventionandthemannerinwhichtheinterventionisbeing,orhasbeen,implemented.

1InternationalFederationofRedCrossandRedCrescentSocieties(2011).Project/programmemonitoringandevaluation(M&E)guide,availablefrom:http://www.ifrc.org/Global/Publications/monitoring/IFRC-ME-Guide-8-2011.pdfandAustralianDevelopmentAgency(2009).GuidelinesforProjectandProgrammeEvaluations,FinalDraft,availablefrom:https://www.oecd.org/development/evaluation/dcdndep/47069197.pdf.

www.appro.org.af 6

Thedatatobeincludedinabaselineassessmentisafunctionofthegoalstheinterventionaimstoaccomplish,thetheoryofchangefortheintervention,andtheindicators(seebelow)tobeusedtomonitortheinterventionandevaluateitsimpact.Thedatacollectedforabaselineassessment,andthesubsequentmonitoringandevaluation,mayincludeofficialstatistics,existingsurveyresults,andreliableresearchreports,journals,andnewspaperarticles.Itmayalsobenecessarytocollectnewdatainsituationswheretheexistingdataisoutdatedorwhereoperatingconditionsarecontinuallychanging.Incaseswherenewdataarerequired,thebaselineassessmentshouldundertakesituationanalysis,stakeholderanalysis,resourcemapping,formalsurveys,interviewsandfocusgroupdiscussions.2Datacollectedduringmonitoringandevaluationmustdirectlycrossreferencetheindicatorsdefinedthroughconductingthebaselineassessment.Abaselineassessmentisthusacrucialelementofanymonitoringandevaluationframework.ObjectivesSpectrum:Aprojectorprogramisdesignedandimplementedtomeetaseriesofobjectivesatdifferentlevels,fromtheveryhighlevelandgeneral,i.e.thegoalofaproject/program,totheverylowlevelandspecific,i.e.,theinputsrequiredtoimplementaproject/program.Figure1depictstheobjectivesspectrum,highlightingareasoffocusformonitoringandevaluation.Figure1.ObjectivesSpectrum

Source:IFRC(2011:11)Indicators:Qualitativeandquantitativeevidenceofmovementtowardorawayfromproject/programobjectivesatdifferentlevelsofobjectives(seeObjectivesSpectrum,above).Anindicatorprovidesareasonablysimpleandreliablebasisforassessingachievement,change,andperformance.Indicatorsarenormallydevelopedatthestartoftheprojectcycleandshouldbethedirectoutputofabaselineassessmenttogaugethestartingpoint(seeBaselineAssessment,above).

2UNWomen(2017),“Whatisabaselineassessment?”availablefrom:http://www.endvawnow.org/en/articles/1323-what-is-a-baseline-assessment.html

www.appro.org.af 7

Monitoringismostlyfocusedonthelowerlevelobjectives,i.e.,inputs,activities,andoutputswhileevaluationscanfocusonanyoralllevelsofobjectives(Figure1).Scope:Thebreadthofmonitoringorevaluationvariesandcanbeencompassanentireproject/programoronlyapartorpartsofaproject/program.Monitoringandevaluationcanbeappliedtoanactivity,setofactivities,project,program,strategy,policy,ororganization.

PurposeofMonitoringandEvaluation

Theaimofconductingmonitoringandevaluationistoprovideinformationthatcanhelpinformdecisions,improveperformance,andachieveplannedresultsthrough:• Providingconsolidatedinformationthatcouldbeanalyzedtodemonstrateprogressin

implementingactivities,projects,programs,orpolicies• Providingameansforagenciesseekingtolearnfromtheirexperiencesandtoincorporatinglearning

intofuturephasesofpolicyandpractice• Generatingreportsthatcontributetotransparencyandaccountabilityoftheinterveningbodyand

allowingforlessonstobesharedmorewidely• Allowingactorstolearnfromeachother’sexperiences,buildingonexpertiseandaddingtothebody

ofknowledge• Addingtoretentionanddevelopmentofinstitutionalmemory• Providingabasisforquestioningandtestingassumptions• Providingameansofassessingthecruciallinksbetweendecisionmakers,implementers,and

beneficiaries• Providinganempiricalbasisfordemonstratingsuccess,failure,andlearning,influencingpolicy,and

raisingfunds.3

DifferencesBetweenMonitoringandEvaluation

Findingsfrommonitoringrevealthecurrentsituationatanygiventimeandovertimerelativetotargetsandoutcomes,andareusedtomodifytheinputsandactivitiesonanongoingbasistoensurethattheproject,program,orpolicyreachesitsintendedgoaloroutcome.Evaluationsarecarriedouttoarriveatconclusionsabouttherelevance,effectiveness,efficiency,impactandsustainabilityofaninterventionatacertainpointduringthelifetimeoftheproject,program,orpolicyorattheend.Theresultsofevaluationsareusedtotakecorrectiveactionforremainderoftheproject,program,orpolicyor,ifcarriedoutattheend,toinformdecisionsonfutureinterventions.

3Adaptedfromsportanddev.org,availablefrom:www.sportanddev.org/en/toolkit/monitoring-and-evaluation/why-monitoring-and-evaluation-me-important

www.appro.org.af 8

Monitoring

Monitoringmaybeinput-andactivity-orientedor“results-based.”Theselectionoftheorientationformonitoringisafunctionofwhatthemonitoringentitywishestoaccomplish,theefficiencyofresourceuseandeffectivenessinimplementingactivitiesortheoutcomesandlongertermimpactoftheintervention.Inanyevent,amonitoringsystemmayhaveoneormoreofthefollowingelements.

InputMonitoring

Monitoringoffinancial,human,andmaterialresourcesrequiredforproducingthedesiredoutputs.

ProcessMonitoring

Monitoringthetransformationprocessthroughwhichresourcesareturnedintofinishedgoodsand/orservices.Thistypeofmonitoringpaysparticularattentiontomonitoringthetaskscarriedoutbythepersonneltotransforminputstooutputs.

OutputMonitoring

Monitoringthegoodsand/orservicesproducedimmediatelyafterthecompletionofthetransformationprocess.

OutcomeMonitoring

Monitoringoftheeffectsofoutputsonthetargetareaorbeneficiaries.

ImpactMonitoring

Monitoringthelongerterm,systemicchangeinsocietyasaproductoftheoutcomesoftheintervention.

ImplementationMonitoring

Focusesoninputs,processesandoutputs,andistypicallycarriedoutforprojectswhicharesmallerinsizeandnotnecessarilydirectlylinkedtoabroader,multi-facetedstrategy.Typically,implementationmonitoringhasthefollowingfeatures:

• Descriptionoftheproblemorsituationbeforetheintervention• Benchmarksforactivitiesandimmediateoutputs• Datacollectiononinputs,activities,andimmediateoutputs• Systematicreportingonprovisionofinputs• Systematicreportingonproductionofoutputs• Directlylinkedtoadiscreteintervention(orseriesofinterventions)• Designedtoprovideinformationonadministrative,implementation,andmanagementissuesas

www.appro.org.af 9

opposedtobroaderdevelopmenteffectivenessissues.Theguidingquestionsfortraditionalmonitoringandevaluationfocusedonwhetherornot:

• Adequateandsufficientinputsweremobilized• Agreeduponactivitieswerebeingcarriedoutandcompleted• Agreeduponoutputs(goodsand/orservices)werebeingdelivered

Results-basedMonitoring

Tendstobemoresystemicandsystematic,coveringawiderrangeofareasforexaminationbyhavingthefollowingfeatures:

• Baselineassessmenttodescribetheproblemorsituationbeforetheintervention• Indicatorsforoutcomes(basedontheinformationfromthebaselineassessment)• Datacollectiononoutputsandwhetherandhowtheycontributetomeetingintervention

objectives• Establishingperceptionsofchangeamongstakeholders• Systemicreportingthroughcollectionandanalysisofqualitativeandquantitativedata,paying

particularattentiontomovementtoward,oraway,frominterventionoutcomes• Involvingthestrategicpartnersoftheintervention• Establishingsuccessorfailureofpartnershipstrategyinachievingdesiredintervention

outcomes.Theguidingquestionsareexpandedtoestablishwhetherornot:

• Outputshaveresultedin(preferablymeasurable)outcomesconsistentwiththeinterventionobjectives

• Outcomeshavemanagedtohavebroadersystemicimpactresultingin,forexample,achangeofattitude

• Impactsaresustainablebeyondthelifetimeoftheintervention.

IndicatorsforMonitoring

Thevalueofamonitoringsystemininformingdecisionsonprojectorprogramimplementationisadirectfunctionoftheindicatorsusedinmonitoring.Abaselineassessmentisavaluablesourcefordefiningusefulandusableindicatorstoreportontheperformanceofaprojectorprogram.Indefiningindicators,caremustbetakentoselectindicatorsforwhichdataisavailableandaccessible.Thetestofusefulnessandusabilityofindicatorisbasedon“SMART”nessoftheindicators,consistingofthefollowingcharacteristics:4

4ThesedefinitionsandexamplesaretakenfromSavetheChildren,availablefrom:https://sites.google.com/site/savethechildrendme/Home/smart-indicators

www.appro.org.af 10

Specific:Theindicatorhastobespecific.Itmustbeabletobetranslatedintooperationaltermsandmadevisible.Whiletheoutcome/resultitselfcanbebroad,theindicatorshouldbenarrowandfocusonthe‘who’and‘what’oftheintervention.Additionally,‘how’and‘where’the‘who’isdoingthe‘what’isimportanttoincludeintheindicatorasitprovidestheactionfortheintervention.

Example:"Parents(who)willdemonstratepositiverelationshipswiththeirchildren(what)intheirhomeenvironment(where)byanincreaseinnurturingcommunication,andappropriateandresponsivecare(how)atthetimeofexitingtheprogram(when)."

Measurable:Theindicatorshouldbemeasurable,thatis,ithasthecapacitytobecounted,observed,analyzed,tested,orchallenged.Ifonecannotmeasureanindicator,thenprogresscannotbedetermined.Howwilloneknowiftheoutcomehasbeenachieved?Onceanindicatorisclearandspecific,theycanbemeasuredinnumerousways;almostanyindicatorisinonewayoranother,measurable.

Example:X%ofparentsserved(measurable)willdemonstratepositiverelationshipswiththeirchildrenintheirhomeenvironmentbyanincreaseinnurturingcommunication,andappropriateandresponsivecareatthetimeofexitingtheprogram.

Achievable:Theindicatorisachievableiftheperformancetargetaccuratelyspecifiestheamountorlevelofwhatistobemeasuredinordertomeettheresult/outcome.Theindicatorshouldbeachievablebothasaresultoftheprogramandasameasureofrealism.Thetargetattachedtotheindicatorshouldbeachievable.

Example:85%ofparentsserved(attainable?)willdemonstratepositiverelationshipswiththeirchildrenintheirhomeenvironmentbyanincreaseinnurturingcommunication,andappropriateandresponsivecareatthetimeofexitingtheprogram.

Relevant:Anindicatormustberelevant.Itshouldbeavalidmeasureoftheresult/outcomeandbelinkedthroughresearchandprofessionalexpertise.Thereisnoreasontocreateanindicatorwhichdoesnotrelatetothelargeroutcome.Theindicatorshouldbemeaningfulandimportanttotheoutcometocertifythattheresultsareactuallyshowingarelatedimpact.Broadoutcomes/resultscanandshouldhavenumerousspecificandapplicableindicatorsthroughwhichprogresscanbeassessed.Anindicatorisrelevanttotheextentthatitcapturesormeasuresafacetoftheoutcomethatitisintendedtomeasure.Thebestwaytothinkaboutrelevanceistoensurethatthereisarelationshipbetweenwhattheindicatormeasuresandthetheoriesthathelpcreatetheoutcomesfortheclient,program,orsystem.Thebestmethodtofindrelevantindicatorsistoconsultexpertinputandproperresearch.Thematerialculture,asitiscalled,typicallydiscussesindicatorsforcertainphenomena;forexamplechilddevelopment,familyfunctioning,andchildabuseprevention.Time-bound:Theindicatorisattachedtoatimeframe.Theindicatorshouldstatewhenitwillbemeasured.Ifthereisnotimeincludedonwhentomeasuretheindicator,howwillanyoneknowifandwhenthereisaresult/outcome?

Example:85%ofparentsservedwilldemonstratepositiverelationshipswiththeirchildrenintheirhomeenvironmentbyanincreaseinnurturingcommunication,andappropriateandresponsivecareatthetimeofexitingtheprogram(timebound).

www.appro.org.af 11

UseofIndicatorsinPolicyMaking

SMARTindicatorsusedinmonitoringplayanimportantroleinthepolicycycle,particularlyincapturingstakeholderviews,facilitatingmoreopengovernanceoftheintervention,andenablinginformeddecisionmaking.Tounderstandtheuseofindicatorsintheprocessofdecision-making,weneedtofirstlookatthecycleofpolicy-making.Policy-makinginvolvespolicyplanning,implementation,evaluationandlearningandadjustment.Indicatorsplayakeyrolebyhelpingtooutlinepolicygoalsinspecificterms,allowingmonitoringtheprogress,evaluatingagainstsetcriteria,andprovidingfeedbacktomanagersandthepublicaboutoutcomes(Figure2).Figure2:ThePolicyProcess

Source:Parto,S.(2015)5Assumingthatastraightforward connection between specific policies and outcomes can be made,indicatorscanplayakeyroleincontinuouspolicylearningandadaptation.Apolicy intentorgoalis formulated into a policy statement. The policy is then taken up,interpreted, and applied to the target group. Its character and strength are influenced by thedegreeofunderstandingwithwhichitisappliedandthevigorwithwhichitisenforced,monitored,andevaluated.

5Parto,S.(2015),“PolicyAnalysisandInstitutionsofGovernance:Analyzing….What?”availablefrom:http://appro-europe.net/publication/policy-analysis-institutions-governance-analyzing-what/

www.appro.org.af 12

Evaluations

Evaluationsmaybecarriedoutperiodicallyorattheendofaprojectorprogram.Periodically,evaluationmaybeundertakenattheendofsetperiods,e.g.,annually,throughthelifetimeofaprojectorprogram.Typically,forprojectsandprogramsexceedingtwoyearsthereareamid-termandanend-termevaluation.

EvaluationEthics

Thefollowingethicalprinciplesshouldbeadheredtoinalltypesofevaluation.

ObjectivityofEvaluators

Everyevaluationneedstoachieveamaximumlevelofobjectivityandimpartialityinitsdesignandimplementation.6Statementsoffactneedtobemethodicallyandclearlyidentified.Differentperspectivesandstrengthsandweaknessesmustallbeaccountedfor.Allfindingsandrecommendationsmustbesupportedbyevidenceandmustbecomprehensible.

IndependenceofEvaluators

Theevaluatorsmusthaveexpertknowledge,becredible,andunrelatedtothepersonneloftheproject/programbeingevaluated.

Participation

Anevaluationneedstobeasparticipatoryaspossiblethrough,forexample,jointdevelopmentofthetermsofreferencewiththeprojectpartnersandcreationofopportunitiesforallrelevantpartiestocommentontheresultsoftheevaluationandtheevaluationreport.

TransparencyAndFocus

Theevaluationassignmentmustbeclearlydefinedandfocused.Theremustbecleardescriptionsoftheprojectorprogram,objectivesoftheevaluation,centralquestions,methodology,qualificationsoftheevaluationteam,andreportingrequirements.Inmostcases,anevaluationcannotcoverallOECD/DACevaluationcriteriatothesameextent.Assuch,astrictdefinitionofthescopeandprioritiesoftheevaluationisessential.

Reliability

Theutilizationandpreparationofbasicdataisnecessaryinordertoprovetheassessmentandtheconclusionsinacrediblefashion.Theevaluationresultsstatedintheevaluationreportmustbecomprehensible.

CompletenessAndClarity

ThereporthastobestructuredaccordingtotheOECD/DACcriteriaandevaluationquestions.All

6ThissectionisadaptedfromAustrianDevelopmentAgency(2009),“ADAEvaluationBooklet-GuidelinesforProjectandProgrammeEvaluations”,pages9-10,availablefrom:https://www.oecd.org/development/evaluation/dcdndep/47069197.pdf

www.appro.org.af 13

evaluationquestionsmustbeanswered.

FairnessAndProtectionofPartiesInvolved

Security,dignityandrightsofthepersonsinvolvedintheevaluationmustbeprotected.

Utility

Evaluationrecommendationsareusedforimprovingprojectsorprograms.Feedbacktopoliticalandoperativedecisionmakersmustbeguaranteedthroughaclearresponsibilityfortheimplementationoftheevaluationresults.Theutilityofanevaluationalsodependsonthemostappropriatemomentoftheimplementationwithintheprojectorprogramcycle

EvaluationCosts

Theexpenditureoftimeandfinancialmeansofanevaluationneedtobeinareasonableproportiontoitsbenefitsandthescopeoftheprojectorprogram.Normally,about3–5percentofthetotalprojectorprogrambudgetisshouldbespentonevaluations.

EvaluationCriteria

Therehavebeenstandardizedcriteriaforevaluationssincethelate1950s,focusingmainlyoneducationalassessmentandconductedbysocialresearchersandsomeacademicinstitutions.Startinginthemid1960sintheUnitedStates,thefederalgovernmentmadeevaluationsofpastsocialandeducationalprogramsasthebasisofitsdecisionmakingonreleasingnewfundsfornewprograms.Bythe1990stherewasincreasedemphasisbygovernmentsandvarioustypesoforganizationtouseevaluationasthemeanstobecomemoreefficientandeffectivewhilelearningfrompastexperienceinasystematicmannertoinformfuturedecisionmakingonprogramming.Sincethe2000stheemphasisinevaluationshasshiftedtoparticipation,collaborationandlearning.7Paralleltothesedevelopments,sincetheearly1960stheOrganizationforEconomicCooperationandDevelopment(OECD)hadbeguntoemphasizethenecessityandvalueofcriticalevaluationsindevelopmentaidasameanstoincreaseefficiency,effectiveness,andlearning.8AseriesofconferencesorganizedbyOECDinthe1970sresultedintheformationofa“GroupofCorrespondentsonAidEffectiveness”,taskedto:

1. Reportonexistingfindingsonaideffectiveness2. Considerfeedbackintopolicy-making3. Reportonwaysinwhichevaluationcouldbesupportiveofpublicinformation,and4. Considerhowtosupportevaluationworkindevelopingcountries.

OECD’sDevelopmentAssistanceCommittee(DAC)publishedthe“PrinciplesofProjectAppraisal”in1988,thefirstconcretesteptowardstandardizingthemannerinwhichevaluationofaidprogramming

7SeePreskill,H,andD.Russ-Eft(2005),“BuildingEvaluationCapacity:72ActivitiesforTeachingandTraining”(SagePublications).

8TheinformationinthissectionisdrawnfromOECD(2013),“TheDACNetworkonDevelopmentEvaluation:30yearsofstrengtheninglearningindevelopment”,availablefrom:http://www.oecd.org/dac/evaluation/Eval%20history%20booklet%20web.pdf

www.appro.org.af 14

wastotakeplace.InsubsequentyearsDACproducedseveralprinciplesandgoodpracticesfortechnicalassistance,programassistance,womenindevelopment,environmentalassessment,andprocurement.Thesewerepublishedtogetherin“DevelopmentAssistanceManual,DACPrinciplesforEffectiveAid”in1992.DAC’sfinalized“CriteriaforEvaluatingDevelopmentAssistance”areRelevance,Effectiveness,Efficiency,Impact,andSustainability(Box1).9Box1:DACEvaluationCriteriaRelevance:Theextenttowhichtheaidactivityissuitedtotheprioritiesandpoliciesofthetargetgroup,recipientanddonor.Inevaluatingtherelevanceofaprogrammeoraproject,itisusefultoconsiderthefollowingquestions:

• Towhatextentaretheobjectivesoftheprogrammestillvalid?• Aretheactivitiesandoutputsoftheprogrammeconsistentwiththeoverallgoalandtheattainmentof

itsobjectives?• Aretheactivitiesandoutputsoftheprogrammeconsistentwiththeintendedimpactsandeffects?

Effectiveness:Ameasureoftheextenttowhichanaidactivityattainsitsobjectives.Inevaluatingtheeffectivenessofaprogrammeoraproject,itisusefultoconsiderthefollowingquestions:

• Towhatextentweretheobjectivesachieved/arelikelytobeachieved?• Whatwerethemajorfactorsinfluencingtheachievementornon-achievementoftheobjectives?

Efficiency:Efficiencymeasurestheoutputs–qualitativeandquantitative–inrelationtotheinputs.Itisaneconomictermwhichsignifiesthattheaidusestheleastcostlyresourcespossibleinordertoachievethedesiredresults.Thisgenerallyrequirescomparingalternativeapproachestoachievingthesameoutputs,toseewhetherthemostefficientprocesshasbeenadopted.Whenevaluatingtheefficiencyofaprogrammeoraproject,itisusefultoconsiderthefollowingquestions:

• Wereactivitiescost-efficient?• Wereobjectivesachievedontime?• Wastheprogrammeorprojectimplementedinthemostefficientwaycomparedtoalternatives?

Impact:Thepositiveandnegativechangesproducedbyadevelopmentintervention,directlyorindirectly,intendedorunintended.Thisinvolvesthemainimpactsandeffectsresultingfromtheactivityonthelocalsocial,economic,environmentalandotherdevelopmentindicators.Theexaminationshouldbeconcernedwithbothintendedandunintendedresultsandmustalsoincludethepositiveandnegativeimpactofexternalfactors,suchaschangesintermsoftradeandfinancialconditions.Whenevaluatingtheimpactofaprogrammeoraproject,itisusefultoconsiderthefollowingquestions:

• Whathashappenedasaresultoftheprogrammeorproject?• Whatrealdifferencehastheactivitymadetothebeneficiaries?• Howmanypeoplehavebeenaffected?

Sustainability:Sustainabilityisconcernedwithmeasuringwhetherthebenefitsofanactivityarelikelytocontinueafterdonorfundinghasbeenwithdrawn.Projectsneedtobeenvironmentallyaswellasfinanciallysustainable.Whenevaluatingthesustainabilityofaprogrammeoraproject,itisusefultoconsiderthefollowingquestions:

• Towhatextentdidthebenefitsofaprogrammeorprojectcontinueafterdonorfundingceased?• Whatwerethemajorfactorswhichinfluencedtheachievementornon-achievementofsustainability

oftheprogrammeorproject?

9BasedonOECD(1986),“GlossaryofTermsUsedinEvaluation,in'MethodsandProceduresinAidEvaluation'”;OECD(1991),“TheDACPrinciplesfortheEvaluationofDevelopmentAssistance”;andOECD(2000),“TheGlossaryofEvaluationandResultsBasedManagement(RBM)Terms.”

www.appro.org.af 15

Indesigningevaluations,itisimportanttobeclearaboutscope(seeDefinitions,above).Withthescopeestablished,thefollowingstepsneedtobetakentocarryouttheevaluation:• Initiate“entranceconferences”toensureallrelevantstakeholdersareawareoftheevaluationbeing

carriedoutandareavailableandwillingtocollaboratewiththeevaluatorsthroughouttheevaluationprocess.

• Prepareadatarequestlistandsubmittotheevaluationcommissioningbody.• Reviewlogframe,projectreports,previousevaluations,andotherdocumentedinformation

pertainingtotheproject.• Reviewannualbudgetsandexpenditurereports,budgetguidelinesandprocedures,accounting

procedures,salaryschedules,organizationcharts,staffingratios,strategic(orequivalent)plans,performancereports,andotherrelevantdocuments.

• UsingtheDACEvaluationCriteria(Box1),organizeinterviewswithkeystakeholdersincludingprojectstafffromtheevaluationcommissioningbody,implementingorganizations,representativesampleofbeneficiaries,relevantgovernmentauthorities,andothersasappropriate.

• Organizeprojectsitevisits.• Carryoutanalysisofqualitativeandquantitativedatafromsecondarysourcessuchasreportsand

primarysourcessuchasinterviewandfocusgrouptranscriptsandsurveyforms.• Presentthepreliminaryfindingsbasedontheanalysisofallavailableinformationtotheevaluation

commissioningbody.• Incorporatefactualcorrections,ifany,intothefindings.• Finalizetheevaluationreport.