Evaluation Matters - How can we Strengthen National ......• Strengthening National Evaluation Systems Samuel Kouakou, Côte d’Ivoire • The importance of Strengthening National

3

5

7

28

29

76

90

From the Director’s iPadRakesh Nangia

Defining Capacity and Evaluation Capacity Development

Towards a Shared Framework For National Evaluation Capacity Development

Perspectives from Around the Globe

Africa •Côted’Ivoire •Ethiopia •SouthAfrica •Uganda •CLEARInitiativeforFrancophoneAfrica •AfricanDevelopmentBank

Latin America

Food for Thought • HowtoSpeedupGlobalDevelopment’s

LearningCycle • DevelopingEvaluationCapacity,butWhich

Capacity? • ChecklistforDevelopingaNational

EvaluationSystem

In This Issue:

September 2013

eVALUatiOn Matters

How can we Strengthen National Evaluation Systems?

—Perspectives from around the globe

“It’s a bit like the Titanic: Do you want to course correct and miss the iceberg, or do you want to set up a comprehensive review of why it hit the iceberg and what lessons you could learn?”

—Trevor Davies in How to Speed Up Global Development’s Learning Cycle

A Quarterly Knowledge Publication of the Operations Evaluation Department of the African Development Bank Group

A QUARTERLY KNOWLEDGE PUBLICATION OF THE AFRICAN DEVELOPMENT BANKSeptember 2013 . Volume 2 . Number 3

3 From the Director’s iPad Rakesh Nangia

5 Defining Capacity and Evaluation Capacity Development

7 Towards a Shared Framework For National Evaluation Capacity Development

Marco Segone, Caroline Heider, Riitta Oksanen, Soma de Silva, and Belen Sanz

26 Framework for Understanding NECD Definitions and Terminology

28 Perspectives from Around the Globe

29 Africa• StrengtheningNationalEvaluationSystems

Samuel Kouakou, Côte d’Ivoire• TheimportanceofStrengtheningNationalEvaluationSystems

F. Kelemework Fasika, Ethiopia• DevelopingaNationalEvaluationSysteminSouthAfrica,

Ian Goldman, South Africa• PatternsandInfluencesintheSupplyandDemandofEvaluation

andMonitoringinUganda’sPublicSectoroverthePastTwoDecades David Rider Smith, Uganda• StrengtheningNationalEvaluationSystems

El Hadji Gueye, Senegal• TowardsanEvaluationCultureintheAfDB:Practicesand

Challenges Mohamed Manai, AfDB• Rebirth,Restoration,Reclamation,andResponsibilitiesofthe

EvaluationFunctionofAfrica Sukai Prom-Jackson, Joint Inspection Unit of the United Nations System

76 Latin America• SpeakingTruthtoPower:TheRoleofNationalEvaluationSystems

inDemocraticGovernance Dr. Claudia Maldonado,General Coordinator, Clear Center for Spanish-Speaking Latin America

90 Food for Thought• HowtoSpeedupGlobalDevelopment’sLearningCycle

Trevor Davies, KPMG• DevelopingEvaluationCapacity,butWhichCapacity?

Samer Hachem, AfDB• NationalEvaluationCapacityDevelopment:

PastandFutureDirections Mohamed Manai, AfDB• ChecklistforDevelopingaNationalEvaluationSystem

evaluation Matters

is published quarterly in English and French

Publications Coordinator: Felicia Avwontom

Produced under the guidance of the

Director of OPEV, Rakesh Nangia.

Contributors: Mohamed Manai, Jessica

Kitakule, Samer Hachem, Pierre-Joseph

Kingbo, Deborah Glassman and Caroline

McEuen (consultants—provided editorial

assistance)

Design: Phoenixdesignaid and Felicia

Avwontom (AfDB)

layout and Printing: Phoenixdesignaid

Cover photo: OPEV Learning Event, SARC

© 2013—African Development Bank (AfDB)

African Development Bank Group

15 Avenue du Ghana,

Angle des rues Pierre de Coubertin et Hedi

Nouira BP 323, 1002 Tunis Belvédère

Tunisia

The mission of the Operations Evaluation Department is to enhance the

development effectiveness of the AfDB

in its regional member countries through

independent and instrumental evaluations

and partnerships for sharing knowledge

Director: Rakesh Nangia, [email protected]

Managers: Samer Hachem, [email protected]

Mohamed Manai, [email protected]

Questions? Telephone: (216) 71 102 841

Fax: (216) 71 194 460

Web: http://operationsevaluation.afdb.org

Write to us: [email protected]

Copyright: © 2013—African Development

Bank (AfDB)

eVALUatiOn Matters

2 eVALUatiOn Matters

3A quarterly knowledge publication of the Operations Evaluation Department of the African Development Bank Group

Rakesh Nangia, Director, Operations Evaluation Department, AfDB

Fiscal pressures, greatly magnifiedbytheglobalfinancialcrisis,brought not only pressure on officialdevelopmentassistancesupportbutalsoincreaseddemandsfromexternalstake-holders.Thefocusonresultsisunprec-edented,andgoeshand-in-handwiththedemandforuseable,results-basedmonitoringandevaluation(M&E)systemstoprovidetangibleevidenceofoutcomes,aswellassupportforevidence-baseddecisions.

Thetraditionalrolefortheindependentevaluationfunctionhousedinvariousmul-tilateraldevelopmentbanksistofocusontheperformanceofprogramsfinancedbytherespectiveinstitutions.Mostoftheevaluationsconductedareex-post.However,regardlessofhowwelltheyareconductedandhowinfluentialtheyarewithrespecttoreforms,theyareasmallpartofthenationalpublicinvestmentprogram.

Itisobviousthatinfluencingtheentire investmentprogramisthesmarterwaytogo.Inordertodoso,developmentinstitutions,includingtheirrespectiveindependentevaluationdepartments, needtorethinktheirtraditionalrole.BuildingcapacityinM&Ehasbeenpart,albeitasmallpart,oftheprogramofthesedepartmentsformanyyears.Butcapacityisonlyoneelementofthesystem,whichincludestrainingstaff;establishingappropriateorganizationalstructures;ensuringgood-qualitydataandflow;and,ofcourse,usingtheevidencetoinformdecisions.Whatroleshouldtheevaluationdepartmentsplayinbuildingthesesystems?Isitevenwithintheirmandates?

Iwouldsubmitthatcontinuedcapacity-buildingeffortsintheabsenceofastrengthen-ingoftheremainingelementsoftheoverallsystemisawastedeffort.Thevalueofa

From Experience to Knowledge … From Knowledge to Action … From Action to Impact

Evaluations Matters: From the Director’s iPad


holisticperspectiveisoneofthelessonsthatvariousevaluationreports(oncapacitybuild-ing)havehighlightedovertheyears.Yetwe,theevaluationcommunitythatconstantlyadvocatesforlearninglessonsfromexperience,seemtoignorethisadviceinourowncapacity-buildinginitiatives.

Ideally,weshouldbeginwithananalysisoftheexistingM&Esystem,includingidentify-ingthegaps.Thisshould,ofcourse,bedonewithcounterpartsingovernments,whichonewouldhopeincludessomechampions.Sharingtheexperiencesofothercountries,includingOECDmembers,wouldbeanintegralpartofthegapsanalysisandaddressingsystemcon-straints.FocusingondataqualityandflowisasimportanttotheefficacyoftheM&Esystemastrainedstaff.Theinterdependenceofthepillarsofthesystemisimportanttosuccess,andsup-portingthemtogetherinaholisticwayiskeytosustainableresults.

Fortunately, policymakers are recognizingthe importance of addressing these issues

holistically.Tanzania,wherePresidentKikwetehaslaunchedtheBIGResultsNowprogram,isagoodexample.Theprogram,whichwaslaunchedlessthan oneyear ago,isinspiredbyasimilar,successfulprograminMalaysia.Asmallresultssecretariat,reportingdirectlytothepresident,consolidatesresultsfromeachofthelineagencies,whichalsohaveensconcedresultssecretariatsoftheirown.Eachofthelineagenciesisresponsibleforcollectingtheresults-focuseddatafromthe districts andreportingtothelineagencysecretariat.Thepresidenthasasignedresultscommitmentwitheachminister.Thiscommitmentismadepublic,soaveragecitizenshaveapotentialinstrumentforholdingtheirgovernmentaccountable.

Allofthisisexpectedtoinfluencethepublicinvestmentprogram,andthusaperformance-andresults-basedbudgetallocation.Itisclearlyveryearlydays,buttheoverarchingvisionanddirec-tionarelaudable.Maybe,justmaybe,theevalu-ationcommunitywilllearnfromtheexperienceandplayacatalyticroleinSouth-Southlearningandscalingupacrossthedevelopingglobe.

Ideally, we should begin with an analysis of the existing M&E system, including identifying the gaps. This should, of course, be done with counterparts in govern-ments, which one would hope includes some champions.

Rakesh Nangia is the Director of the Operations Evaluation Department of the African Development Bank. Prior to joining the AfDB, he spent 25 years at the World Bank, where he held several positions includ-ing Director of Strategy and Operations for the Human Development Network and Acting Vice-President for the World Bank Institute.

He attended the Indian Institute of Technol-ogy in Delhi and Harvard University and holds degrees in business administration and engineering.

Defining Capacity and Evaluation Capacity

Development


the united nations Evaluation Groupdefinescapacityasthe

abilityofindividuals,institutionsandsocietiestoperformfunctions,

solveproblems,andsetandachieveobjectivesinasustainablemanner,

andcapacitydevelopmentastheprocessthroughwhichtheabilitiesto

dosoareobtained,strengthened,adaptedandmaintainedovertime.

ForOECD,evaluationcapacitiesaretheabilityofpeopleandorganiza-

tionstodefineandachievetheirevaluationobjectives(OECD,2006).

Capacityinvolvesthreeinterdependentlevels:individual,organizational

andtheenablingenvironment.Theseinterrelatedcapacitiesfunction

togethertodemand,supplyanduseevaluation.Evaluationcapacities

includethepowertosettheevaluationagendaandtodeterminewhatis

evaluatedandwhatquestionsareasked(OECD,2011).Further,capacities

includetheabilitynotjusttoproduceevaluationsbutalsotomanage

evaluationprocessesandeffectivelyuseevaluationresultstoinfluence

policyandprogrammedecisions.Itisusefultodistinguishbetweenthe

capacitytomanageevaluationsandthecapacitytoconductthem,as

botharenecessary(Feinstein,2009).

EvaluationCapacityDevelopment(ECD)isunderstoodastheprocess

ofunleashing,strengtheningandmaintainingevaluationcapacities.

(excerptedfromTowardsaSharedFrameworkforNationalEvaluation

CapacityDevelopment,page10).

Defining Capacity and Evaluation Capacity

Development


Putt

ing

eval

uat

ion

to w

ork


Marco Segone, UNICEF Evaluation Office, Co-chair EvalPartners, Co-chair UNEG Task Force on National Evaluation Capacity Development

Caroline Heider, Director General and Senior Vice-President, Independent Evaluation Group, World Bank

Riitta Oksanen, Ministry for Foreign Affairs, Finland; Chair OECD/DAC Task Force on Evaluation Capacity Development

Soma de Silva, President, IOCE, and Co-chair EvalPartners

Belen Sanz, Chair, UNEG, and Head, Evaluation Office, UN Women

This paper was published by EvalPartners in the book Evaluation and Civil Society.

Reprinted here by permission

IntroductionNational ownership and leadership as overarching factors for ensuring relevant development outcomesTheParisDeclaration onAid Effectivenessendorsedin2005,andthe2008Accrafollow-upmeeting,statethatnationalownershipandleadershipareoverarchingfactorsforensuringgooddevelopmentoutcomes.Theimplicationfortheevaluationfunctionisfundamental.

Theprincipleofownershipmeansthatpart-nercountriesshouldownandleadtheirowncountry-ledevaluationsystems,whiledonorsandinternationalorganizationsshouldsup-portsustainablenationalevaluationcapacitydevelopment.The2012BusanHigh-levelforumreaffirmedtheaboveprinciples,whilerecogniz-ingthattheinternationaldevelopmentarenahaschangedsignificantly.Asaresult,newmodalitiessuchassouth/southandtriangularcoopera-tion,andnewstakeholderssuchasCivilSocietyOrganizations(CSOs),havebeenexplicitlyrec-ognizedintheBusan’s“Partnershipforeffectivedevelopmentcooperation”.TheBusanprinci-plesarealsosupportedbytheUnitedNations.TheUNGeneralAssemblyhasrequestedtheUNsystemtopursueandintensifyitsefforts

1 Thisarticleisbasedon:a)Segone,M.(2010)editor.FromPoliciestoResults:DevelopingCapacitiesforCountryMonitoringandEvaluationSystems.UNICEF,DevInfo,IDEAS,ILO,IOCE,UNDP,UNIFEM,WFPandWorldBank,andb)Heider,C.(2011).ConceptualFrameworkforDevelopingEvaluationCapacities.Building onGoodPractice. In: Influencing Change. Building Evaluation Capacity to Strengthen Governance.TheWorldBank.

Towards a Shared Framework For National Evaluation Capacity

Development1


to strengthen evaluation capacities in pro-gramcountries,takingintoaccountnational

conditionsandensuringrespectfornationalownership,strategiesandsovereignty.

National evaluation capacities as part of good governanceAscountriestakegreaterownershipof,andleadershipin,theirdevelopmentprocesses,theyhavealsoincreasinglydevelopedtheirsystemsto lead,manage and account for resourcesinvestedintheseprocessesandresultspro-ducedwiththem.Results-basedbudgetingandmanagementhavebeenpartoftheagendatostrengthengovernanceandgohandinhandwithcapacitiesformonitoringandevaluation.Evaluation capacities empower stakehold-ers—fromnationalgovernmentstoCSOs—toquestion,understandandtakechargeofnecessarychangestodevelopmentprocessesasandwhenevidenceindicatespolicies,pro-gramsandprojectscanbemoresuccessful,

effectiveandefficient.Exercisingevaluationinanindependent,credibleandusefulwayisessentialtorealizethecontributionitcanmaketogoodgovernance,includingaccountabilityfromgovernmentstotheircitizensandtheirdevelopmentpartners,transparencyintheuseofresourcesandtheirresults,andinlearningfromexperience.Understandinghowpowerfulevaluationcanbeinmakinginformeddeci-sionswill leadtothecreationofanenablingenvironmentanddemandforevaluationasanintegralpartofdebatesanddecision-makingprocesses,andwillstimulatedevelopmentorstrengtheningofevaluationsystemstomeetthesedemands.

Strengthening existing national evaluation capacitiesInanumberofcountries,publicpolicyevaluationfunctionsareinplace(UNEG,2011).Examples

arethosesetupthroughtheimplementationofPovertyReductionStrategyPapers,whichlinkmonitoringandevaluationsystemstopovertyreduction.SeventypercentofcountriesinAfricahavethistypeofsetupwithdifferentinstitutionshavingthemandateforevaluation.Thereisaposi-tivemovetowardsindependenceoftheevaluationfunctioninseveralcountries.AgrowingnumberofcountrieshaveincludedtheevaluationfunctionintheirConstitutionormandateditviaActsofParliament.Insomecountriestheinvolvementofcivilsocietyisgainingmomentumthroughdialogueandpeerreviewmechanisms.

However,whilemanygovernmentshavedefinedevaluationfunctions,andputinplaceinstitu-tionalarrangementsforevaluationtodeliverevi-dencetoinformpolicy-making,existingcapacityvariesverysignificantlyfromcountrytocountry.

In a number of coun-tries, public policy evaluation functions are in place. Examples are those set up through the implementation of Poverty Reduction Strategy Papers, which link monitoring and evaluation systems to poverty reduction.


Thismeansthatanyinitiativetostrengthennationalevaluationcapacitiesshouldrecog-nizeexistingcapacitiesandstrengthenthem,

beingsensitive todifferent levelsofcapac-ity,ratherthanstimulatingnewandparallelcapacities.

The growing role of civil society organizations in national evaluation capacity developmentAlongthelinesoftherecentBusanDeclaration,CSOscanandshouldplayacentralroleinadvo-catingfortransparencyintheallocationandexpenditureofpublicbudgets;accountabilityfortheimplementationofpublicpolicies;strength-eningthedemandanduseofevaluationtoinformevidence-basedpolicy-making;and,strengthen-ingcapacitiesofqualifiedevaluatorstoproducecredibleandusefulevaluationsbasedonnationalandinternationalevaluationstandards.

While national evaluation capacity is oftenunderstoodasgovernmentcapacity,itshouldbeacountry-basedcapacity,includingVoluntaryOrganizations for Professional Evaluation2 (VOPEs),universities,thinktanksandNon-governmentalOrganizations(NGOs).CSOsandparliamentariansneedevaluationstosupporttheirunderstandingofissuesandparticipationindecisionmaking.Thiscanstrengthenthequal-ityofdemocracywherebyinformedcitizensareabletoinfluencedecision-making.

Inthelastdecades,membersofbothcivilsoci-etyandtheprivatesectorhavebeenplayingincreasinglycentralandactiverolesinpromot-inggreateraccountability forpublicactionsthroughevaluation.

NationalandregionalVOPEsgrewfrom15inthe1990stomorethan150nowadays3.

In this context, 25 organizations launchedEvalPartners4,aninternationalcollaborativeinitiativetocontributetotheenhancementofthecapacitiesofCSOs—notablyVOPEs—toinfluencepolicy-makers,publicopinionandotherkeystakeholderssothatpublicpoliciesareevidence-based,equitableandeffective.ThemainobjectiveistoenhancethecapacitiesofCSOs/VOPEstoengageinastrategicandmean-ingfulmannerinnationalevaluationprocesses,tobeabletoinfluencecountry-ledevaluationsystems.

Towards a shared framework for national evaluation capacity developmentIn addition to governments, VOPEs andNGOsatcountrylevel,amultitudeofstake-holders is currently engaged in supportingNationalEvaluationCapacityDevelopment:

2 Sinceanumberofdifferentnamesareusedtodescribethesegroups, the term Voluntary Organizations for ProfessionalEvaluation(VOPEs)hasbeenintroduced.VOPEsincludeformallyconstitutedassociationsorsocieties,aswellasinformalnetworksandcommunitiesofpractice.Theirmembershipsareopennotonlytothosewhoconductevaluationsbutalsotothosewhocommissionandutilizeevaluationsandthoseengagedinbuildingtheevaluationfield.

theUnitedNationsEvaluationGroup(UNEG)anditsmembers,theEvaluationCooperationGroup(ECG)oftheInternationalFinancialInstitutions(IFIs),the Development AssistanceCommittee (DAC) of theOrganisation forEconomic Cooperation and Development

3 IOCEmaintainsadatabaseofVOPEsonitswww.IOCE.netwebsite,includinganinteractiveworldmapshowingthelocationsofandcontactinformationfornationalandregionalVOPEs.4 Pleasevisithttp://www.mymande.org/evalpartnersforaddi-tionalinformation.


(OECD),theregionalCLEARcenters,amongothers.However, in certain cases differentactorshavenotcoordinatedtheirsupportandinitiatives,makingitdifficulttocreateposi-tivesynergiesandefficiency.Inextremecases,duplicationsandparallelprocessesaresup-portedbydifferentagencies.Therefore,there

isaneedtocreatesynergiesbasedonasharedframeworkforNationalEvaluationCapacityDevelopment.Thiswillhelpguidedevelopmentactivitiesinacomprehensivewaytostrengthennationalevaluationsystemsasawhole,inaddi-tiontoprovidingguidanceongoodpractice,basedonevidenceofwhatworksandwhy.

Defining Capacity and Evaluation Capacity DevelopmentThe United Nations Development Group(UNDG)definescapacityastheabilityofindi-viduals,institutionsandsocietiestoperformfunctions,solveproblems,andsetandachieveobjectivesinasustainablemanner,andcapacitydevelopmentastheprocessthroughwhichtheabilitiestodosoareobtained,strengthened,adaptedandmaintainedovertime.

ForOECD,evaluation capacities aretheabil-ityofpeopleandorganisationstodefineandachieve their evaluation objectives (OECD,

2006).Capacityinvolvesthreeinterdependentlevels:individual,organisationalandtheena-blingenvironment.Theseinterrelatedcapacitiesfunctiontogethertodemand,supplyanduseevaluation.Evaluationcapacitiesincludethepowertosettheevaluationagendaandtodeter-minewhatisevaluatedandwhatquestionsareasked(OECD,2011).Further,capacitiesincludetheabilitynotjusttoproduceevaluationsbutalsotomanageevaluationprocessesandeffec-tivelyuseevaluationresultstoinfluencepolicyandprogramdecisions.Itisusefultodistinguishbetweenthecapacitytomanageevaluationsandthecapacitytoconductthem,asbothareneces-sary(Feinstein,2009).

Evaluation Capacity Development (ECD)isunderstoodas theprocessofunleashing,strengthening and maintaining evaluationcapacities.Capacitydevelopmentisalong-term,endogenouschangeprocessthattakesplaceinthecontextofongoingpartneranddonoreffortstostrengthenrelatedsystemsofmanagement,governance, accountability and learning, toimprovedevelopmenteffectiveness.Thebestcapacitydevelopmentapproachesareflexible,adaptedandsustainable.

Strengtheningevaluationcapacitiesisnotanendgoalinitself,butshouldbeseen,rather,

The United Nations Development Group (UNDG) defines capacity as the ability of individuals, institutions and societies to perform functions, solve problems, and set and achieve objectives in a sustainable manner, and capacity development as the process through which the abilities to do so are obtained, strengthened, adapted and maintained over time…For OECD, evaluation capacities are the ability of people and organizations to define and achieve their evaluation objectives.


asameanstosupportmoreeffectivedevelop-mentactivitiesandinformedpolicy-making.Evaluationimplicatesactorsacrosstheentire

rangeofstakeholdersinvolvedinandaffectedbypublicpolicy:governments,intendedben-eficiaries,civilsociety,andthegeneralpublic.

A Systems Approach to National Evaluation Capacity DevelopmentNational Evaluation Capacity Development(NECD) is a complex field inwhich severalstakeholdershavedifferentrolestoplaybasedontheirrespectivevalueadded.Thiscomplex-ityencouragestheuseofasystemsapproachtoNECD.Thismeansthatitisnecessarynotonlytolookatactorsatdifferentlevelsandacross

sectorsbutalso,crucially,tolookatthenetworkofrelationshipsorconnectionsbetweenthem.Suchaviewpointillustratesthefactthatweaknessesincapacityatanylevelorwithanykeyactorwillaffectthecapacityofthewholesystemtodealwithaprobleminordertoachieveagoal.Therefore,asystemsapproachtoNECDisneeded.

Individual and institutional evaluation capacities enabled by a supportive environmentInthepast,evaluationcapacitydevelopmentfocusedonstrengtheningthecapacitiesofindi-viduals’knowledgeandskills.However,itisbynowclearthatcapacitydevelopmentshouldbebasedonasystemicapproachthattakesintoaccountthreemajorlevels(individual,institu-tional,andexternalenablingenvironment),and

twocomponents(demandandsupply5)tailoredtothespecificcontextofeachcountry.

5 “Supply”referstothecapabilityofprofessionalevaluatorstoprovidesoundandtrustworthyevaluativeevidence;“Demand”referstothecapabilitybypolicymakersandseniormanagerstorequestsoundandtrustworthyevaluativeevidencewiththeaimofusingitinstrategicdecision-makingprocesses

Figure 1: A systemic and integrated approach to national evaluation capacities development

Individual Level

EnablingEnvironment

InstitutionalLevel

Equity-focused and Gender-responsive

Dem

and S

up

ply

– Tailored to the specific context of each country– Country leadership and ownership


Enabling Environment: the enabling environment provides a context that fosters (or hinders) the performance and results of individuals and organizations

•Strongevaluationculture: •evaluative(critical)thinking; •protectiveenvironment; •managersandotherdecision-makersvalueanduseevaluation.•Publicadministrationcommittedtotransparencyandmanagingforresultsandaccountability,

throughresults-basedpublicbudgetingandevidence-basedpolicymaking.•Politicalwilltoinstitutionalizeevaluation.•Existenceofadequateinformationandstatisticalsystems.•Legislationand/orpoliciestoinstitutionalizemonitoringandevaluationsystems.•Dutybearers,i.e.Governmentsandnationalauthorities,havecapacityandwillingnesstobe

accountableforresults.•Rightsholdersi.e.citizensandCSOs,havecapacitytodemandthatpolicyandprogramsare

monitoredandevaluated.•NationalVOPEexists,hasthenecessarycapacities,isrecognizedandgivenaconsultative

role.•Nationalevaluationstandardsandnormsdevelopedandimplemented.

Institutional Level: the institutional framework in which individuals work needs to provide a system and structures to perform and attain results individually and collectively, as an organization

•Evaluationpolicyexistsandisimplemented.•Anevaluationunitwithaclearlydefinedrole,responsibilitiesandanindependentplacein

theinstitutionalstructureexistsandisfunctional.•QualityAssurancesystemexistsandisfunctional.•Independenceoffundingforevaluations.•Adequate(number&expertise)staffingoftheevaluationfunction.•Systemtoplan,undertakeandreportevaluationfindingsinanindependent,credibleand

usefulwayexists.•Systemtoestablishaformalmanagementresponsetotherecommendationsofevaluations

feedingbacktopoliciesandprogrammes,follow-upofimplementation,exists.

Towards a shared framework for National Evaluation Capacity Development

•Opendisseminationofevaluationresults.•Knowledgemanagementsystemsinsupportoftheevaluationfunctionexistsandisused.•Technicalcompetenceforconductingevaluationsexist.


Individual Level: the individual whose knowledge, skills and competences are essential to perform tasks and manage processes and relationships

•Seniormanagementcapacityto: –strategicallyplanevaluations; –assurethedevelopmentofrelevantandappropriateTermsofReferenceforevaluations; –manageevaluationforindependenceandcredibility; –promotetheuseofevaluationfindings; –followuprecommendations.•Atmid-managementlevel,understandingoftheroleofevaluationasatoolforeffectively

achievingdevelopmentresults.•Identifyandsupportleadersornaturalchampions.•Behavioralindependenceandprofessionalcompetencesofthosewhomanageand/orconduct

evaluations.•Promotecapacitydevelopmentandinvolvementofagrowingnumberofevaluators.

Source:AdaptedfromSegone,M.(2010),Movingfrompoliciestoresultsbydevelopingnationalcapacitiesforcountryledmonitoringandevaluationsystems.

Theenabling environment for evaluation isdeter-minedbyacultureoflearningandaccountabil-ity,meaningthedegreetowhichinformationissoughtaboutpastperformance,theextenttowhichthereisadrivetocontinuouslyimprove,andtoberesponsibleoraccountableforactionstaken,resourcesspent,andresultsachieved.Suchcultureisembeddedintacitnormsofbehavior,theunderstandingofwhatcanandshould—orshouldnot—bedone,inmanycasesbehaviorsbeingrole-modeledbyleaders.Throughasetofvaluesandattitudessupportingevaluative(critical)thinkingwithinanorganization,indi-vidualsaremoreself-directedlearnersanduseinformationtoact;totakehigherrisksbut,alsotodevelopagreatersenseofpersonalaccount-abilityandresponsibility;and,toconsult,coach,andsupporteachothermore.Inthiscontext,organizationswithacultureofevaluationareabletodevelopinnovativeideasandstrategies;changemorequicklyaccordingtovariationsintheexternalenvironment;and,increaseeffi-ciencyandeffectivenessbysystematicallyusing

lessonslearnedtoimproveprogramsandpoli-cies.Thereislessdirectionfromtopmanage-mentandamuchmorepositiveattitudeandself-accountabilityatallorganizationallevels.Anorganizationwithacultureofevaluationhasaneffective,structuredandaccepteduseofevaluationtosupportchangeanddevelopment.Managersvalueanduseevaluationfindingstotestoutinnovationorassessprogresstowardsexpectedresults.Thesetacitnormsofbehav-iorsare,orshouldbecodifiedingovernmentlegislationand/oranevaluationpolicythatexpressestheleadership’soranorganization’scommitmenttolearning,accountabilityandtheevaluationprinciples.Atwo-tierstrategyshouldbeputinplacetostrengthenthecapacityofdutybearers(policy-makers)todemandandusesoundevidencewhiledevelopingrights-holders’(i.e.citizens,CSOs,etc.)capacitytodemandandtoassesspolicyimplementation,byputtinginplacesystemsandmechanismstoengagecitizengroups,andtocaptureandutilizetheirfeedback.


An enabling environment is also supportedorcreatedthroughgovernancestructuresthatdemandindependentevaluation,beitthroughparliamentsorgoverningbodies,andisfurtherenhancedthroughVOPEsthatsetstandardsandstrivetowardsgreaterprofessionalisminevalu-ation.Therefore,VOPEsshouldbesupportedtoenablethemtofosterindigenousdemandandsupplyofevaluation,includingbysettingnationalevaluationstandardsandnorms.TherearealsoexamplesofgovernmentssolicitingtheadviceandinvolvementofVOPEsinnotonlytheformulationofevaluationpoliciesandsystems,butalsointheimplementationofevaluationsconsistentwiththosepolicies(PresidencyofSouthAfrica,2011).

Thestructuralindependenceofanevaluationfunctionisimportanttosetanenablingenvi-ronment:byplacingtheevaluationfunctioninsuchawaythatitisnotcontrolledbythepersonorfunctionresponsibleforthepolicies,strate-giesoroperationsthatareevaluated,andinawaythatcreatesanenvironmentthatenablesgreaterindependence,credibilityandutility.Ideally,theenablingenvironmentissuchthatdecision-makersproactivelydemandimpartialevaluationstoinformtheirdebatesandchoices,whichincreasestheusefulnessofevaluations.

The institutional framework for evaluationensuresthatasystemexiststoimplementandsafeguardtheindependence,credibilityandutil-ityofevaluationwithinanorganization.Suchaninstitutionalframeworkhasthefollowingcharacteristics:

• Includes a system of peer review orassur-ancethattheevaluationfunctionissetuptosafeguardandimplementtheprinciplesofindependence,credibilityandutility.

• Establishes safeguards to protect individual evaluators—evaluators,evaluationmanagers,andheadsofevaluationfunctions—whenexercisingtheir independence, includingtransparentandcredibleprocessesfortheselection,appointment,renewal(ifapplica-ble)andterminationofcontractoftheheadofevaluationandassurancesthatevaluationstaffdonotsufferinpromotionexercises.

• Ensures a multidisciplinary team existsintheevaluationfunction,oronanevaluationteam,thatensurescredibilityofevaluationbyunderstandingmultipledimensionsofevalu-ationsubjectsandcombiningthenecessarytechnicalcompetence.

• Secures the independence of funding of evalu-ations,atanadequatelevel,toensurethatnecessaryevaluationsarecarriedoutandthatbudgetholdersdonotexerciseinap-propriate influenceor controloverwhatisevaluatedandhow.Therefore,fundingshouldbeunderthedirectcontroloftheheadoftheevaluationfunction,andshouldbeadequateforareasonableworkprogram.Theadequacyoffundingforevaluationcanbedeterminedthrougharatioofevalua-tionresources(humanandfinancial)overthetotalprogramsize(someorganizationsaimtoallocate1to3percentoftheirtotalbudgetforevaluation)orthecoveragerateofevaluations(forinstance,evaluating25percentoftheoverallprogram).

• Combines measures for impartial or purpo-sive selection ofevaluationsubjectstoensureimpartiality,ontheonehand,andincreasedutility,ontheother,bymakingdeliberatechoiceslinkedtodecision-makingprocesses.


Toensureutility,forinstanceofstrategicorthematicevaluations,thesemaybechosenpurposively(ratherthantoestablisharep-resentativesample)tolinktheconductoftheevaluationtotheinformationneedsanddecision-makingprocessesofthoseusingevaluationfindings.Inthesecases,aconsul-tationprocesstodeterminewhatthemostimportant/strategictopicsareforevaluationisimportanttoensurevariousstakeholderneedsareconsidered.

• Sets out a system to plan, undertake and report evaluation findings in an independ-ent, credible and useful way.To increaseobjectivityintheplanningandconductofevaluation,systemsareneededthatincreasetherigor,transparencyandpredictabilityofevaluationprocessesandproducts.Suchsystemscanincludemoreorlessdetailedprocessdescriptionsorguidelinesforthedesignofevaluations,forpreparatoryworkthatneedstobeconductedandreportedon,andforreportingfindings.Theprocessesshouldhavebuilt-instepsforcommunica-tion,consultationandqualityassurance,andbecommunicatedtostakeholderstoenhancetransparencyandtosecuretheirwillingnesstoshareinformation.

• Institutes measures that increase the useful-ness of evaluations, includingthesharingoffindingsandlessonsthatcanbeappliedto other subjects. Evaluations should beundertakenwiththeintention—ofstake-holdersandevaluators—tousetheirresults,andmanagement’sresponsesdevelopedandimplemented.Thetimelinessofplanningandconductingevaluationsandpresent-ingtheirfindingsisequallyimportantto

ensureutility.Finally,accessibilitymeansthatevaluationsareavailabletothepublic,canberetrieved(forinstancethroughauser-friendlywebsite),arewritteninwaysthatareunderstandable(clearlanguageandlimitedjargon),andaredistributedtoawidegroupofrelevantstakeholdersinformatsappropriatetodifferentaudiences.

Evenwithstructuresandsystemsinplace,theindependenceandimpartialityofevaluationdependsontheintegrityandprofessionalismof individualsasevaluators,evaluationmanag-ers,andtheheadofevaluation.Theprofession,anditsethics,requireslimitingpersonalbiasestotheextentpossible.Insomecircumstances,external evaluators are believed to exercisegreaterindependencethanthosewhoworkinanorganization,becausetheyarelessexposedtoinstitutionalorpeerpressuresorhavenotabsorbedthecorporatecultureinanunques-tioningway.However,individualorintellectualindependenceisdependentontheindividual,whodemonstratesinhis/herbehaviortheadher-enceto,andpracticeof,theevaluationprinciples:avoidingconflictofinterest;actingwithintegrityandwithanindependenceofmind;engaginginevaluationsforwhichtheyarecompetent;actingimpartially;and,undertakinganevalu-ationwithaclearunderstandingoftheclientsordecisionmakingprocessandhowtheseneedtobeinformed(seeAEA,2003).ManyVOPEsandevaluationfunctionsofnationalandinter-nationalorganizationshaveadoptedcodesofconductforevaluators.Inadditiontothese,theUNEvaluationGroupdevelopedethicalguide-lines(UNEG,2005).Debatesaboutprofessionalstandardsandtheaccreditationofevaluatorsandevaluationmanagershavebeenongoingformanyyearswithinprofessionalforums.


Development of national evaluation capacityAninstitutionalframeworktoinstitutionalizetheevaluationfunctionshouldbedevelopedwithintheorganization.Thisshouldincludeassistancetodevelopanevaluationpolicy,whichforeseesthesetting-uporstrengtheningofacredibleevaluationdepartment,includingendorsementofevaluationstandardsandqualityassurancesystems(includingpeerreview).Adiagnosisoftheexistingevaluationfunctionand/orsystem(or,ifnotexisting,ofthepreconditionstodevelopit),whichwouldincludefunctionalclarity,effec-tivehumanandfinancialresourcesmanage-mentandarobustcoordinationmechanism,shouldbecarriedouttoenablethecraftingofacontext-specificevaluationpolicyandsystem.

Thediagnosticwouldidentifybottlenecksinthepolicyorsystemwhichcouldbeaddressedthroughtargetedtechnicalsupportwith theintentionofstrengtheningcountrycapacity.

Agoodpracticeistostrengthenknowledgeman-agementsystemsinsupportoftheevaluationfunc-tion.Knowledgehastraditionallybeenfosteredattheindividuallevel,mostlythrougheducation.However,seenfromtheperspectiveofthethreelevelsidentifiedabove,knowledgeshouldalsobecreatedandsharedwithinanorganizationthroughaneffectiveknowledgemanagementsys-tem,andsupportedthroughanenablingenviron-mentofeffectiveeducationalsystemsandpolicies.

Figure 2: Tips for stakeholders to help reinforce an enabling environment for evaluation

Governing bodies (Parliaments, Executive Boards, etc.)

•Beawareofthethreatstoindependence,credibilityandutilityofevaluationanddemandmeasurestosafeguardtheseprinciples.

•Adoptandoverseetheimplementationoflegislationand/orpolicies,whichinstitutionalizetheindependence,credibilityandutilityofevaluation.

•Putinplaceeffectiveoversightmechanismsoverthequalityofevaluation.•Useevaluationfindingsandrecommendationsinnationalandsubnationalpolicies,programs

andlegislation.

Heads of State, Ministers, Policy-Makers, Chief Executives Officers

•Demonstrateleadershipinsettingacultureoflearningandaccountability.•Seekanduseevidencefromevaluationstovalidatetheattainmentofgoalsandobjectivesand

toimproveperformancewheneverpossible.•Understandevaluationaspartofgoodgovernancethataimstousepublicresources.Effectively

andefficientlytoachievethegoalswhichgovernmentsororganizationsaimtoachieve.


Evaluation Function

•Demonstratethevalueaddedofindependent,credibleandusefulevaluations.•Raisetheawarenessofstakeholdersoftheroleandimportanceofevaluationandoftheevalu-

ationprinciples.•Contributetoevaluativethinkingthroughawarenessbuilding,dialogue,andtraining.•Defineandbuildevaluationcompetenciesatdifferentlevelsofpublicoffice.

VOPEs

•Workwithgovernmentstosetstandardsasbenchmarkswhichcanbeusedtoconvinceotherstakeholdersoftheimportanceoftheevaluationprinciplesandmeasurestosafeguardthem.

•Fosterindigenousdemandandsupplyofevaluation.•BeavailabletoadvisecommissionersofevaluationsontherelevanceofTermsofReference

forevaluations,includingchoicesofappropriatedesignsandmethodologiestoanswerkeyquestions.

•Promotethecapacityofevaluatorstoperformquality,credibleandusefulevaluations.•Advocateforequity-focusedandgender-sensitiveevaluationsystems.

Source:AdaptedfromHeider,2011,(Conceptualframeworkfordevelopingevaluationcapacities).

Attheindividuallevel,acapacitydevelopmentstrategyshouldstrengthensenior management capacity to strategically plan evaluations(andtoidentifythekeyevaluationquestions);tomanageevaluationforindependenceandcredibility;and,to use evaluation.

MacKay(2007)underlinestheimportanceofidentifying and supporting leaders or natural champions whohavetheabilitytoinfluence,inspireandmotivateotherstodesignandimple-menteffectiveevaluationsystems.Leadershipisnotnecessarilysynonymouswithapositionofauthority;itcanalsobeinformalandbeexer-cisedatmanylevels.Therefore,theevaluationcapacitydevelopmentstrategyshould,especially

inthe initialstages, identifyandsupportasappropriate,nationalandlocalleadersinthepublicadministration,ininter-governmentalmonitoring, and in evaluation groups andnationalVOPEs.

Onthesupplyside,acapacitydevelopmentstrat-egyshouldenhancebehavioral independence (independenceofmindandintegrity;knowledgeandrespectofevaluationstandards;agreedeval-uationprocessesandproducts)aswellasprofes-sionalcompetencesthroughformaleducation;specializedtraining;professionalconferencesandmeetings;onthejobtraining(suchasjointcountry-ledevaluations);and,communitiesofpracticeandnetworking,e.g.VOPEs.


Figure 3: Tips for stakeholders to help develop an institutional framework for evaluation

Governing bodies (Parliaments, Executive Boards, etc.)

•Bebriefedabouttheevaluationsystemsoastounderstandwhethertheinstitutionalframeworkincludesadequatechecksandbalancesandtobecomeadiscerninguserofevaluationreports.

•Introduceprocesses for the selectionof theheadof evaluationwhich ensureshis/herindependence.

•Providesecure,separateandadequatefundingforevaluationintheregularprogrambudgets.

Head of State, Ministers, Policy-makers, Chief executives officers

•Bebriefedabouttheevaluationsystemsoastounderstandwhethertheinstitutionalframe-workincludesadequatechecksandbalances,andtobecomeadiscerninguserofevaluationreports.

•Establishorstrengthenevaluationtrainingatgraduateandundergraduatelevelstoprovideprofessionalevaluatorsinappropriatequantities.

•Createprofessionalevaluationpositionsinsupportoftheevaluationfunctioningovernmentbodies.

Evaluation function

•Developanddocumentsystemsfortheselection,design,conductandreportingonevaluations.•Providebriefingsonthesestandardstoincreasetransparencyandconfidenceintheprocess

andproducts,andsotoenhancecredibility.•Developmechanismstoensurelessonsfromevaluationaresystematicallysharedandinte-

gratedintodebatesanddecision-makingprocesses.

VOPEs

•Provideguidancetosetprofessionalstandardsandgoodpracticestandards.•Assistthegovernmenttodevelopandimplementprofessionalization,accreditation,andcre-

dentialingsystems.

Source:AdaptedfromHeider,2011,(Conceptualframeworkfordevelopingevaluationcapacities).


Fostering demand for and supply of evaluationAdistinctionshouldbemadebetweenthecapacityofpolicy-makers/policyadvisorstouseevidenceandthecapacityofevaluationprofessionalstopro-videsoundevidence.Whileitmaybeunrealisticforpolicy-makers/policyadvisorstobecompetentexpertsinevaluation,itisbothreasonableandnecessaryforsuchpeopletobeabletounderstandanduseevidenceproducedbyevaluationsystemsintheirpolicyandpractice.Integratingevidenceintopracticeisacentralfeatureofpolicy-makingprocesses.Anincreasinglynecessaryskillforpro-fessionalpolicy-makers/policyadvisorsistoknowaboutthedifferentkindsofevidenceavailable;howtogainaccesstoit;and,howtocriticallyappraiseit.Withoutsuchknowledgeandunderstandingitisdifficulttoseehowastrongdemandforevidencecanbeestablishedand,hence,howtoenhanceitspracticalapplication.

However,itisalsoimportanttotakeintocon-siderationthatthedesignandimplementationofpolicyreformisapoliticalprocess,informedbyevidence.Theuseofevidenceinpolicyreformandimplementationdependsonthecombina-tionofcapacitytoprovidequalityandtrustwor-thyevidenceontheonehand,andthewilling-nessandcapacityofpolicy-makerstouseitontheother.Theextenttowhichevidenceisusedbypolicy-makersdepends,inturn,onthepolicyenvironment.

Tostrengthenanenablingpolicyenvironment,policy-makersmayneedincentivestouseevi-dence.Theseincludemechanismstoincreasethe‘pull’forevidence,suchasrequiringspendingbidstobesupportedbyananalysisoftheexist-ingevidence-base,andmechanismstofacilitateevidence-use,suchasintegratinganalyticalstaffatallstagesofthepolicyimplementation.

Civilsocietyorganizations,includingVOPEs,shouldplayamajorroleinadvocatingfortheuseofevidenceinpolicyimplementation.Think-tanks,withthesupportofmassmedia,mayalsomakeevidenceavailabletocitizens,andcitizensmaydemandthatpolicymakersmakemoreuseofit.

Itisthereforeclearthatastrategyfordevelopingnationalevaluationcapacitiesshouldbetailoredtothesituationandcontextinwhichaspecificcountryfindsitself,asillustratedinFigure4,anddetailedbelow:

• Vicious circle countries.Evidenceistech-nicallyweakandpolicymakershavelittlecapacitytomakeuseofit.Policydecisionsaremainlytakenonpoliticaljudgmentandpersonalexperience,whichmayresultinpoorpolicydesignand,consequently,poorresults.Inthiscase,itisnecessarytoadoptmeasureswhichwillsimultaneouslyincreaseboththedemandandsupplyofevidence,aswellasimprovethedialoguebetweenpro-ducersandusersofevidence.

• Evidence supply-constrained countries. Althoughevidenceistechnicallyweak,itisincreasinglydemandedbypolicy-makers.However,evidencedeficiencyreducesthequalityofdecision-makingandthereforethequalityofservicesdelivered.Policymakersarelikelytoresentbeingheldtoaccountonthebasisofinadequateevidence.Therefore,thepriorityshouldbetoadoptmeasurestoincreasethequantityandqualityofevi-dence.Thechallengeistostrikeabalancebetweenquicklygeneratingimprovementstoevidence,whilelayingthefoundationsfor


Figure 4: Capacity development framework addressing the demand as well as the supply side

Source:AdaptedfromSegone,2009,Enhancingevidence-basedpolicy-makingthroughcountry-ledmonitoringandevaluationsystems

Qu

alit

y an

d t

rust

wo

rth

ines

s o

f ev

iden

ce

Enabling policy environment

Evidence demand-

constrained countries

Practice ofpolitical Life

Timing ofthe analysis

Politicaljudgement

Personalexperience

Hig

hLo

w

HighLow

Financialresources

Lobby system> Think-tank> Opinion leaders> Media> Civil Society

Virtuous circle

countries

Viciouscircle

countries

Evidencesupply-

constrainedcountries

Evidence-informed Evidence-based

Opinion-based Evidence-informed

betterperformanceofthecountryevaluationsysteminthelongrun.

• Evidence demand-constrained countries.Thequantityandqualityofevidenceisimprov-ing,butitisnotdemandedfordecisionmak-ingbecausepolicy-makerslacktheincentivesand/orcapacitytoutilizeit.Policy-makersarelikelytobeattheveryleastwaryof(ormayevenactivelydislike)havingmoreandbetterevidencepushedatthemwhenitmaynotsupportdecisionstheyhavealreadytakenorwishtotake.Inthiscase,priorityshouldbe given to the adoptionofmeasures to

The quantity and quality of evidence is improving, but it is not demanded for decision-making because policy-makers lack the incen-tives and/or capacity to utilize it.


increasethedemandforevidence,aswellastoimprovethedialoguebetweenproducersandusersofdata.

• Virtuous circle countries.Evidenceistechni-callyrobustandisbeingusedincreasinglyfordecision-making.Theproductionofgood(oratleastimproved)evidenceismatchedbyitswidespread(oratleastincreased)useindecision-making.Thesetwoprocessesaremutuallyreinforcing,resultinginbet-terpolicydesignandimplementation,andultimately,betterdevelopmentresults.

Thevirtuouscirclecountriessituationisthegoalwhichshouldbeset,sinceitprovidesausefulbenchmarkagainstwhichtocomparetheotherthreecases.Developingacultureofevidence-basedpolicy-makingisaslowprocesswhichmaytakeyears,butthepotentialrewardsareworththeeffort.Essentialelementsforsustainingavirtuouscirclelinkingevaluationprofessionalstopolicy-makersare:strengtheningthedemocraticprocessbyrequiringtransparencyandaccountabilityinpublicsectordecision-making,andestablishingclearaccountingstandardsandaneffectiveregu-latoryframeworkfortheprivatesector.

Key Principles of National Evaluation Capacity DevelopmentThenumberofstakeholdersandtherangeofcapacitiesandcapabilitiesdiscussedaboveillus-tratethecomplexitiesofcapacitydevelopmentandflagtheneedforplanningandimplemen-tationtoolsthatareabletoaddresscomplexity,somethingthatblueprintstendnottobegoodat.Instead,spaceneedstobecreatedforanalyzingcapacitygapsinaparticipatoryway,agreeingoncommongoalsandtranslatingthemintoajointstrategy.Atthesametimetheflexibilityandcapabilityshouldbemaintainedtotakeupopportunitiesastheyarise,learnfromexperi-enceandchangetactics,andworkonvariouslevelsofcapacityatthesametime.

Theprinciplesofnational ownership and leader-ship havetodowiththerealizationthatcapacitydevelopmentneedstocomefromwithinand/orhavestronginternalchampions,whetheritiswhendevelopinganationalsystemorafunctioninanorganization.Ifanideaisimposedfromtheoutside,chancesarethatcapacitydevelopment

isnotlikelytosucceed.Thedrivefromwithinrelatestothecapability“tocommitandengage.”Itincludes:

a) Ownership,whichmanifestsitselfinhav-inglocalorinternalchampions,resourceallocations,andengagement.Itis,however,difficulttomeasure,nothomogenousacrossinternalstakeholders,andnotconstantovertime;

b) Leadership,whichisimportant,especiallyforsettingthecultureoflearningandaccount-ability,butalsomustfitwiththecontextanditsculture;and

c) Collectiveaction,motivationandcommit-ment,meaningthatcapacitydevelopmentwillnotoccur if it isonepersonwritingdocuments,butrequiresengagingvariousstakeholderswhosecapacitiesandcapabili-tieswillbedeveloped.


Capacity development should therefore beunderpinnedbythefundamentalcharacter-isticofnationalownership.Takingacapacitydevelopmentstrategytoscalerequireslinkingittonationalandlocalplans,processes,budgetsandsystems.Tobesustained,acomprehensivecapacitydevelopmentresponsemustlinkto,anddrawfrom,relevantnationalreforms.

Theaboveisparticularlyimportantascapac-itydevelopmentisabouttransformationsandmustaddresshowbesttomanagechangewithintheexistingpolicyenvironment.Thetendencytooftenlookonlyinsideanorganizationandtodownplaythelargerinstitutionalcontext,inwhichtheorganizationresides,hasprovenunsuccessful.Toensurecontinuedpoliticalcom-mitmentandresourcesupport,acapacitydevel-opmentstrategycanandoftenmustshowbothshortandlong-termgains.Experienceshowsthatcapacitytrapsaremoreoftenpertinenttothe“softside”ofthepolicyenvironment,suchasvestedinterests,ethicandattitudes,ratherthanto“hard”technicalcompetencies.

Theprincipleofcontext-specific capacity devel-opment meansthatcapacitymustbeunder-stoodintermsofaspecificcultural,socialandpoliticalcontext.Capacitymustbeunderstoodassomethingthatexistsindegreesatalllevelsofsociety:individual,household,community,institutionalandnationalsystems.Capacitycanexistwithoutoutsideintervention,thoughitmaybeconstrained.Thisimpliesthatonemustfirstunderstandcapacitiesorelementsofcapacitythatalreadyexistbeforeengaginginanyefforttobuildonorstrengthenthem.

“Context-specific”capacitydevelopmentmeansthatnoblueprintshouldbeapplied,buthigh

degreesofflexibilityshouldbecombinedwithsystematicapproachestoentailarecognitionthatcapacitydevelopment,whilesystematic,doesnot(always)followanorderedprocess.Insteaditincludesplannedaswellasincrementalandemergentapproaches.Ontheotherhand,whentryingtoinfluenceculturetobecomemoresup-portiveoflearningandaccountability,oppor-tunitiesmightariseatshortnotice,invarioussettingsandwithoutpriornotice,andincludeaninformalconversationwithakeystakeholder.Inthesecontexts,capacitydevelopmentini-tiativeswouldbeincrementalandemergentinresponsetoopportunitiesastheyarise.Capacitydevelopmentalsorequiresrecognizingthelesstangibleaspectsofcapacitiesandcombiningsmallandlargeinitiatives,dependingoncontextandopportunities.Itrequiresfindingtherightbalancebetweenanoperatingspacethatallowscapacitiestoevolveandaccountabilityforcapac-itydevelopmentresults;abalancethatishardtoattain.Andfinally,capacitydevelopmenttakestime,soitisnecessarytostaythecourse(eventhroughadversetimes),butbuildquickwinsintotheprocess(tokeepupmotivationandbuildonsuccess),andtimetoreflectandevaluatewhetherprogressisbeingmade.

The process of developing and agreeing onclearexpectationsin“country-specific”capac-itydevelopmentisinitselfpartofthecapacitydevelopmentprocess.Developingacommonunderstandingofcapacityweaknessesthroughaparticipatory,structureddiagnostic—usingtoolsliketheconceptualframeworkinFigure1(p.11)—canserveasananalyticalframeworktomoveattentionfromresourcegapstorecognizebroaderissues,andasharedvisionoftheneededcapacitiesareimportantstepsinchannellingresourcestowardsclearobjectives.Veryoften


themanyactorsincapacitydevelopmenthavetacitunderstandingsofwhatcapacityisneededandhowitsdevelopmentshouldhappen.Fewofthemhaveexplicitstrategies,whichmakesitmoredifficulttofindacommongroundand

workinthesamedirection.Adiagnosisofexist-ingcapacitiesisnecessarytofindentrypointstostartcapacitydevelopment.Thisdiagnosisbenefitsfrombeingdevelopedinaparticipatorywaythatdevelopsownershipatthesametime.

Aligning with national evaluation capacity development strategies and systemsStrongnationalownershipandleadershiparethefoundationforaligningexternalsupportthatleadstosustainableresults.Nationalevalu-ationcapacitydevelopmentstrategiesprovidethestartingpoint.Whenthenationalstrategiesareofhighqualityintermsofhavingclearpri-orities,beingresults-orientedandoperational,theyareanaturalbasisforcooperationandalignment.Anotherimportantindicationofnationalcommitmentistheresourceallocationsintheinstitutions’budgets,andstaffingfortheimplementationofthecapacitydevelopmentstrategies.

Alignmentalsomeansthatcountrysystemsareusedwhenprovidingexternalsupport.Thisincludestheuseofthenational institutions’ownsystemsofplanning,financialmanage-ment,monitoring,reportingandevaluation.Countrysystemsshouldbefullyusedwhentheyprovidereliableassessmentsofperfor-mance,ensuretransparencyandaccountabil-ityofthesystems,andtakenecessarystepstostrengthenandreformthesystems.Providers

ofexternalsupportmayaligntheircontribu-tionstonationalsystemsfullyorpartially.Ifalignmentisonlypartial,thedecisionsshouldberegularlyreconsideredwiththedevelopmentofthecountrysystems.Step-by-stepprogressinalignmentmeansthateffortsandresultsinimprovingcountrysystemsareappropriatelyacknowledged.

Insomecasesthebiggestchallengeforalignmentcomesfromalonghistoryofworkingtogetherinanexternally-ledandsupplydrivenset-up.A recent summaryof lessons learned, fromtheOrganizationforEconomicCooperationandDevelopment’sDevelopmentAssistanceCommittee (OECD/DAC)PeerReviews, onsupportingpartnerstodeveloptheircapacity,callsontheDACdonorsto“takepartnerown-ershipandleadershipseriously”.Thesummaryconcludesthatinsteadoffocusingondonors’domesticaccountabilityandgettingthejobdonewithshorttermresults,achangeofmentalityisnecessaryforlongertermsustainablecapacitydevelopmentresults.

Use of external expert inputs (technical assistance) for demanddriven evaluation capacity developmentCapacitydevelopmentimpliesintentionalitytostrengthencapacities.Forexample,technicalassistancemaystrengthencountrycapacitiesonlyifitisclearlyorientedtowardsthisaim,byengagingcountrystakeholdersintheevaluation

process,creatingspaceforlearningbydoing(i.e.injointcountry-ledevaluations),andfacili-tatingaccesstoknowledge.Capacitydevelop-ment-focusedtechnicalassistanceisthereforedemand-drivenratherthansupply-driven,andit


focusesontheprocessandoutcomesratherthantheinputsoftechnicalassistance.

Guidingprinciplestobeappliedwhenexter-nalexpert inputs (technicalassistance) for

evaluationcapacitydevelopmentareprovidedinclude:

• Technicalassistanceshouldbeaccountabletothenationalinstitutionratherthananexternalstakeholder.

• The Terms of Reference of externallyappointedstaffshouldbedeterminedbythenationalinstitutionjointlywiththeproviderofexternalsupport.

• Thetechnicalassistanceshouldbejointlyselectedandmanagedbypartners.

• Useofnationalandregionalresourcesshouldbepromoted,e.g.throughSouth-Southcooperation.

Figure 5: The shift in paradigm from technical assistance to a capacity development approach

Source:UNDG,2009.ACollectiveApproachtoSupportingCapacityDevelopment.

Technical Assistance

SupplyDriven

Input-based Outcome-based

DemandDriven

CapacityDevelopment

Facilitating accessto knowledge

Facilitating multi-stakeholder engagement

Participatory policydialogue & advocacy

Integrated approachesto local development

Creating space forlearning by doing

(Transformative)

Initiatives and institu-tions for national evaluation capacity development have rapidly increased both in the public sector and in civil society.


Working jointly for evaluation capacity development led by national institutionsInitiativesandinstitutionsfornationalevaluationcapacitydevelopmenthaverapidlyincreasedbothinthepublicsectorandincivilsociety.Therearealsosignsofgrowinginteresttosupporttheseefforts.Theexpandingnumberofpartnershipsforevaluationcapacitydevelopmentbringswithitthetraditionalchallengesofcoordinationandpromo-tionofjointwork.Itisimportantthatnationalinstitutionsexerciseleadershipincoordinatingexternalsupport.Thisimpliesanactiverolefor

thenationalinstitutionsinalldecision-making,includinginjointmanagementstructureswithexternalpartners.Coordination isnecessarytoavoidoverlappingactivitiesandtoensureacomplementaryapproachincomprehensivelysupportingtheevaluationsystem.Cleardivisionoflaborbetweenthepartnersshouldbetheobjec-tive.Workingjointlywithothersshouldalwaysbethepreferredoption,andcarefullyanalyzedbeforelaunchinglessefficientbilateralactivities.

Managing for resultsCapacitydevelopmentmonitoringandreportingoftensufferifthefocusissimplyonwhathasbeendoneandimmediatelyachieved.Reportsontrainingsessionsorganizedandnumbersofpeopletrainedarefamiliartoall.Ashifttoemphasizingmediumandlongertermresultsisnecessary.Theimportantthingistoreportonwhatactuallychangesinindividuals’knowl-edge,skillsandwill,ininstitutionsandintheoperatingenvironmentasaresultofcapacity

developmentefforts.Furthermore,understand-ingwhytheselectedstrategiessucceedorfailincausingchangesistheinputthatisneededforfutureplanning.Thebasisforusefulmonitoringisinhighlevelplanning—clearandmeasur-ableobjectivesaccompaniedwithbaselinedata.Thenationalinstitutions’ownresults-orientedreportingandresultsframeworksshouldbeusedanddeveloped,orcommonarrangementsmadeamongprovidersofexternalsupport.

Mutual accountabilityAllpartnersshouldbeheldaccountablefortheircommitments.Consultationandparticipatoryapproachesbyinstitutionsinplanning,imple-mentation,andassessingprogressarekeystrate-giesforopennessandaccountability.Forexternal

partners,medium-termpredictabilityremainsachallenge.Timely,transparent,comprehen-siveinformationonexternalsupport,includingfinanceflows,isimportantfornationalplanningprocesses.

ConclusionsIn the past, several stakeholders worked tostrengthenNationalEvaluationCapacitiesfocusingontrainingindividuals,ratherthanstrengtheninginstitutionalcapacitiesandanenablingenviron-ment.Inaddition,thankstothechangingexternalenvironmentandtherecognitionandinvolve-mentofnewstakeholders(notablyCSOs,including

VOPEs)andtheneedforbetterpartnershipstosupportnationally-ownedand-ledevaluationsys-tem,thelackofasharedframeworkforNationalEvaluationCapacityDevelopmenthasbecomeevi-dent.Thisarticleisasteptowardsaddressingthisweakness,asafirststeptocreatebetternetworkedpartnership,asintheexampleofEvalPartners.


ExcErptEd from National Evaluation Capacity Development: Practical tips on how to strengthen

National Evaluation Systems information. A Report for the United Nations Evaluation Group (UNEG)

Task Force on National Evaluation Capacity Development

Thereissomeambiguityaboutthekeyconcepts

associatedwithNationalEvaluationCapacity

Development (NECD).Tohelp clarify their

usage,theUNEGdefinesfourconceptsfunda-

mentaltoNECD.Thesecomprise:

(1) thedistinctionbetween ‘evaluation’ and

‘monitoring’;

(2) ‘performancemanagement’and ‘results-

basedmanagement(RBM)’;

(3)‘nationalevaluationsystem’and‘monitoring

andevaluation(M&E)system’,atermoften

usedinitsplace;

(4) in broad terms, what is implied for a

countryby ‘national evaluationcapacity

development’.

Framework for Understanding NECD

Definitions and Terminology

Clarifying the terminology of national evaluation

capacity development‘Evaluation’ versus ‘monitoring’—two tools to measure performance

Theterm‘evaluation’isnotalwayswellunderstood.Itcantakemanyforms,addressarangeof

issuesandpotentiallyserveavarietyofusesanduserstosupportacountry’snationaldevelopm

ent

goals.Itisalsooftenthemostcost-effectivewaytomeasure‘outcomes’associatedwithprojects,

programsorpolicies.

‘Performancemonitoring’canalsoplayanimportantroleinincreasingunderstandingofgovernment

performanceandmeasuringprogressonnationaldevelopmentgoals.Itrequirestheidentification

ofrelevantindicatorsof‘performance’,theestablishmentofongoingmeasurementsystemsandthe

availabilityofcredibledatatopopulatetheindicators.Onceoperational,monitoringinformation

canbeareadysourceofinformationon‘outputs’and(potentially)short-term‘outcomes’,aswell

asprovidinganimportantinputtotheconductofanevaluation.

Monitoringthoughdoesnotprovidethedepthofunderstandingof ‘performance’thatcanbe

gleanedfromanevaluation.Also,evaluationisgenerallyneededtomeasurehigher-levelresults

(‘intermediateandlonger-termoutcomes’and‘impacts’ofpublicsectorinterventions).But,being

adisciplinebasedonsocialscienceresearchmethods,evaluationrequiresahigherskillseta

nd

levelofexperiencethatisoftenlackingorinshortsupplyinacountrycontext.Capacitybuilding

effortsneedtorecognizetheabovedistinctionsandtheimportanceofstrengtheningbothaskey

componentsofaneffectiveandsustainablenationalsystem.

’Performance management’—‘results-based management’

Thesetwotermscanbeconsideredinterchangeable,bothreferringto‘amanagementstrategyfocus-

ingonperformanceandachievementofoutputs,outcomesandimpacts’.Itsbroadobjectiveisto

advancesoundmanagementpracticeandstrengthenaccountabilityandgoodgovernanceacross

allorganizations.

Atacountrylevel,implementingRBMislikelytorequiresomelevelofpublicsectorreform,which

shouldthenbethebroaddriverforNECD,i.e.introducingamoresystematicandresults-oriented

approachtocreatingandusingperformanceinformationingovernmentplanning,budgeting,man-

agementanddecision-making.Itmeansmanagingprogramsandorganizationswithafocusonthe


actualresultsbeingachieved.Todothisrequires:clearlydefiningexpectationsaboutresultst

obe

achieved;deliveringtheprogramsorservices;measuringandevaluatingperformance;and,making

adjustmentstoimprovebothefficiencyandeffectiveness.Italsomeansreportingonperformance

tocitizens.MovementtoaRBMsystem,whetherfororganizationsorcountries,islongtermand

iterative.Thetwotoolsofmonitoringandevaluationthougharecriticalcomponents.

Theterms‘performancemanagement’and‘performancemeasurement’areoftenconfused.The

foundationofperformancemanagement(orRBM)isaccurateandtimelyperformanceinformation.

Thus,NECDeffortsareneededtosupportdevelopmentofaninformationregimethatmeasures,

evaluatesandreportsontheperformanceofprograms;holdsmanagersaccountableforachieving

results;andensuresunbiasedanalysisandreporting.

‘National evaluation system’ or ‘monitoring and evaluation (M&E) system’

Thesearesomewhatambiguoustermsandnotalwayswellunderstood.Whatmaybedeemedto

beacountry’s‘nationalevaluationsystem(NES)’mayinfacthavelittletodowiththepracticeof

‘evaluation’perse.Infact,themorewidelyusedtermfordevelopingcountriesis‘monitoringand

evaluation(M&E)system’andthisisoftenusedinterchangeablywhentalkingaboutacountry’sNES.

Further,inmanyinstances,theprimefocusofthenationalM&Esystemisonthe‘M’,withlittle

ornoinvestmentinthe‘E’.Itisimportant,therefore,toassessandunderstandthecountrycontext

sothatanyNECDeffortsbyUNagenciesrepresentthemosteffectiveinterventionstosupportthe

nationalsystem.ItalsounderscoresthefactthatNECDeffortsinsomecasesmayneedtorecognize

thattheremaybelittleornooperationalcapacityforevaluationattheoutset(thoughcertainly

otherdimensionstobuildupon).

BUT,anationalevaluation(or,M&E)systemneedstobethoughtofasmorethansimplythegenera-

tionof‘performance’information.Evaluation(orM&E)isnotan‘end’initself.Theconceptneeds

tobeviewedintermsofasystemsapproachthatrecognizestheimportanceofbothanabilityto

providesoundevidence(thesupplyside)aswellasthecapacitywithinthesystemforindividuals

andinstitutionstouseinformation(thedemandside).InsupportingNECD,itisthusimportant

toreflecton:(i)whowillbecarryingoutmonitoringandevaluationactivities;and(ii)whowilluse

the‘results’informationacrossthesystem?

National evaluation capacity development (NECD)

Thebasisfor‘nationalevaluationcapacitydevelopment’buildsontheconceptsdescribedabove.

Underlyingitthoughisthebeliefthatresults-basedmonitoringandevaluationrepresentpublic

managementtoolsthatyieldinformationthatcanbeusedtobettermanagepolicies,programsand

projects;demonstrateprogressonnationaldevelopmentgoalstocitizensandstakeholders;and,in

theprocess,toserveaskeyaccountabilitytools.

NECDneedstobethoughtofmorethansimplyintechnocratictermsthough.Nationalownership

impliesaparticularcultural,socialandpoliticalcontext.Thedriverforestablishinganationalevalu-

ation(orM&E)systemoughttobegoodgovernance.And,thiscouldbeassociatedwithgovernment

modernizationinitiatives,theproactiveintroductionofaresults-basedmanagementframework

forthepublicsectorand/orsomeotherformofpublicsectorreform.Thiscouldbetriggeredby

politicalchange;oritcouldbeencouragedbytheactionsandsupportofinternationalagencies

and

donors.Ifitistobeownedbythecountry,itneedstobelinkedtothenationaldevelopmentplan

andintegratedintotheoperationsandcultureofgovernmentinstitutionsandministries.Tobe

sustainablethough,governmentsmustbelieveintheutilityoftheNESandunderstanditsbenefits.

And,todothat,theymusteventuallyownthesystem.


Around the Globe

AfrICA • StrengtheningNationalEvaluationSystems

Samuel Kouakou, Côte d’Ivoire

• TheImportanceofStrengtheningNationalEvaluationSystems

F. Kelemework Fasika, Ethiopia

• DevelopingaNationalEvaluationSysteminSouthAfrica

Ian Goldman • PoliticalPatternsandInfluencesintheSupp

lyandDemandof

Evaluation and Monitoring in Uganda’s Public Sector over the

Past Two Decades, David Rider Smith

• StrengtheningNationalEvaluationSystems

El Hadji Gueye, Senegal

• TowardsanEvaluationCultureintheAfDB:Practicesand

Challenges, Mohamed Manai, AfDB

• Rebirth,Restoration,Reclamation,andResponsibilitiesofthe

Evaluation function of Africa, JSukai Prom-Jackson,

Joint Inspection Unit of the United Nations System

LATIN AMErICA • SpeakingTruthtoPower:TheRoleofNationalEvaluation

Systems in Democratic Governance, Claudia Maldonado

• JointInspectionUnitoftheUnitedNationsSystem

Perspectives From


Samuel Kouakou, Deputy Director of Evaluation and Projects Sustainability, Directorate of Evaluation and Projects, Ministry of Agriculture, Côte d’Ivoire.

Why is it Important to Strengthen National Evaluation Systems?The Paris High-Level Forum on AidEffectiveness, hosted by the Frenchgovernment and organized by theOrganisationforEconomicCo-operationandDevelopment(OECD)inFebruary2005,andtheThirdHigh-LevelForumonAidEffectiveness(HLF-3),heldinAccra,Ghana,September2–4,2008,havegivenarenewedimpetustomonitoringandevaluation(M&E).

Manygovernmentsindevelopingcountriesaredevotingconsiderableefforttostrength-eningtheirmonitoringandevaluation(M&E)systemsandcapacitiestoimprovetheirperformancebyestablishingevidence-basedpolicymaking,budgetdecisionmaking,management,andaccountability.

Why Should We Invest Time and Effort in National Evaluation Capacity Development?Itisimportanttoinvesttimeandeffortinnationalevaluationcapacitydevelopmentbecausecapacitydevelopmentisastrategicinvestmentinthefutureofthecivilservice.Thiscanenhancethewell-beingofcitizensandtheprosperityofthecountry.Itcanalsoimprovegovernmentperformance.1

Weshouldinvesttimeandeffortinnationalevaluationcapacitydevelopmentincivilsociety,becausenongovernmentalactorsarethoughttoplaykeyrolesinboththesupplyofanddemandforinformationandaccountability.

1 UNEG.Practical Tips on How to Strengthen National Evaluation Systems (NewYork,NY:UNEG,2012).

Strengthening National Evaluation Systems - from Côte d’Ivoire


TheestablishmentrecipientM&Eframeworksthatfunctionwell isalsocrucialfordonors,particularlyastheywinddowntheirownparal-lelM&Esystemsandbecomedependentuponrecipientsystems.2

Manylessonscanbedrawnfromexperienceinthedevelopmentofnationalevaluationcapacity.Thesecanbesummarizedasfollows.

Successes• Strongownershipanddevelopment-partner

support fornational evaluation capacitydevelopmentcorereformandsector-basedprograms,theuseofnationalsystems.

• Closecollaborativeapproachbetweendonorsandsectorministries,decentralizedstruc-tures,andwithmunicipalities.

• Efforttofurtherstrengthenperformancebudgetingbygovernments.

• Significant investment in sector-focusedtechnicalcooperationandcapacitydevel-opmentinitiatives.

• Effortstoengagewithcivilsociety.

• Growingnumberofvoluntaryorganizationsforprofessionalevaluation.Indeed,fromonein1981,therearenow135voluntaryorganizationsforprofessionalevaluation.3

2 N.Holvoet,S.Dewachter,M.Gildemyn.Strengthening National Evaluation Societies: An Exploratory Survey,2011(Manchester,U.K.:InstituteforDevelopmentPolicyandManagement,2012).3 J.RughandM.Segone,eds.Voluntary Organizations for Professional Evaluation (VOPEs): Learning from Africa, Americas, Asia, Australia, Europe, and Middle East (EvalPartners, UNICEF, IOCE, 2013), pp. 14–15.

Failures• Theuseofcountrysystems,however,has

remainedvery low,with tone successfulsetup.

• Inmostcountries’nationalM&Esystems,the‘E’forevaluationisessentiallynonex-istentasoneofthetoolstomeasureperfor-mance.Activitiesareessentiallyfocusedonmonitoring.

• M&Eeffortsarestilllargelydonor-driven.

ChallengesThemainchallengefacedinevaluationcapac-itydevelopmentisitsunreliablefundingandresourcebase.Mostcapacity-buildingorganiza-tionsdependlargelyondonorsupporttoconducttheiractivities.Althoughamembershipfeeischargedinsomeorganizationsandgovernmentsupportsinstitutionsorministries,thisconsti-tutesasmallpercentageoftheirannualbudget.Theseorganizationsthereforeexperienceperiodsofverylowactivity,whennofundedprojectisbeingimplemented.

Mostofthevoluntaryorganizationsforprofes-sionalevaluationhaveweaknationalanchoring,andtheyarenotcalleduponbythegovernmentsorbythepartnersindevelopmenttocarryoutevaluationsatanationallevel.Forsuccessfulcapacitydevelopment,wemustensurethatthereisastrongcivilsociety,demandingandadvocat-ingforevaluationvaluesandforevidence-basedpolicymaking.

Anotherchallengeistheproductionofinterna-tionalevaluationstandardsandresources.Thenormsandstandardsproducedbythenetworkalsoserveasaformofdirectcapacitydevelopment


thatprovidespartnerswithresourcestobuildtheirevaluationknowledge.

IttakestimetocreateorstrengthennationalevaluationcapacitiesandtobuildsystemsforsharingM&E information among relevantministries.

Good practices, tipsExtensiveexperiencewithcapacitydevelopmentinitiativeshashighlightedthreemajorlessonslearned:

• Capacity development should be under-pinnedbythefundamentalcharacteristicofnationalownership.Takingacapacitydevelopmentstrategytoscalerequireslink-ingthestrategytonationalandlocalplans,reform,processes,budgets,andsystems.

• Capacitydevelopment is about transfor-mations and must address how best tomanagechangewithintheexistingpolicyenvironment.

• Agoodevaluationcapacitydevelopmentstrategyshouldbetailoredtothespecificcontextofeachcountryandinstitution.Itisveryimportanttounderstandthecountrycontextbeforeproceeding.

Atthelevelofcommunitiesofpractice,wemustshareevaluationexperiences,practices,andinformationatmeetingsorganizedbythosewhohaveaninterestinengaginginevaluationwork.

Thestrategyforstrengthening4M&Esystemsfocusesonthreeaspects:

4 KhadijaKhan.“StrengtheningtheM&ESystem.”(Islamabad:PakistanPovertyAlleviationFund,June2003).

• Takestockofthesituationtoidentifyasce-nariofordetermininggrowingneeds.

• Decideoncriticalstepstobetakenformeet-ingtheaboveneeds,intheformofaplanofactionandimplementingthatplan.

• Definetherolesandresponsibilitiesofpeo-pleatvariouslevelsintheorganizationandbuildcapacityforcarryingoutthoseroleseffectively.5

Examples of models of national evaluation systems that are performing wella) South Africa caseInabout2000,M&EinSouthAfricatookonincreasedprominence.6TherehadbeenM&Eactivitysince1995,butitwasnotcoordinated.In2005,theSouthAfricancabinetapprovedrecommendationsfromthePresident’sOfficeon“animplementationplantodevelopamonitor-ingandevaluation(M&E)system.”Thesystemwastoincludefunctionssuchasmonitoring;evaluation;earlywarning;dataverification;anddatacollection,analysis,andreporting.

TheSouthAfricanConstitutionplacesapre-miumonaresponsive,development-orientedpublicservice,effectiveandaccountablestew-ardshipofpublicresourcesbytheexecutivegovernment,effectiveoversightbyparliamentandthenineprovinciallegislatures,andpub-licparticipationinpolicyandimplementationprocesses.

5 “DesignofMonitoring&EvaluationSystemforStrengtheningthe Sri Lankan Vocational Training System by GTZ SVTPProject—Colombo.”6 IndranNaidoo.Unit:“TheMonitoringandEvaluationinSouthAfrica.ManyPurposes,MultipleSystems.”On-linee-learningprogramoncountry-ledM&Esystems.


AllofthesefactorshaveledtheSouthAfricangovernmenttorecognizetheneedforagovern-ment-wideM&E7system.

b) Australian caseTheelectionofareformistlaborgovernmentin1983providedanenvironmentfavorabletoevaluation.Aseriesofpublic-sectormanagementreformswereimplementedinthefirstseveralyearsofthenewgovernment.Oneaspectofthesereformswasthedesiretolet the managers managebyadevolutionofpowersandresponsi-bilitiesthatreflectedthephilosophythatpublicsectormanagerswouldbestronglyencouragedtoimprovetheirperformanceiftheywerepro-videdgreaterautonomyandthepotentialtomanagetheirdepartmentswithfewercentralagencycontrolsandlessinterference.Thecabi-net’sagreementtotheevaluationstrategy8wasexpressedinaformalcabinetdecision.

7 RonettaEngelaandTaniaAjam.”ImplementingaGovernment-wide Monitoring and Evaluation System in South Africa”(Washington,DC:IEGECDWorkingPaperNo.21,July2010).8 KeithMackay.“TheDevelopmentofAustralia’sEvaluationSystem”(Washington,DC:IEGECDWorkingPaperNo.4,May1998).

Theevaluationstrategyhadthreemainobjectives.Itencouragedprogrammanagerswithindepart-mentstouseevaluationfortheimprovementoftheirprograms’performance.Italsoprovidedfundamentalinformationaboutprogramperfor-mancetoaidthecabinet’sdecisionmakingandprioritization,particularlyintheannualbudgetprocess,whenalargenumberofcompetingpro-posalsareadvocatedbyindividualministers.Andlast,thestrategyaimedtostrengthenaccount-abilityinadevolvedenvironmentbyprovidingformalevidenceofprogrammanagers’oversightandmanagementofprogramresources.

c) Canada caseEvaluationinpublicsectormanagementextendsbackto1969inCanada,9withtheinitiationofformalizedandcentralizedevaluationpractices.Theintroductionofthepracticeofevaluationwasinspiredbythenotionoflettingthemanag-ersmanage;thatis,allowingdeputyministersoffederalgovernmentdepartmentstoassumegreaterresponsibilityfortheirdepartmentsandprograms.Buttheywerealsotobeheldaccount-ablefortheperformanceofthoseprogramsandtheprudentuseofpublicfunds.Themodelisbasedonastrongcentralmanagementboardthatoverseesandholdsdeputiesaccountable,andoneofthemechanismsintendedtodothisistheperformanceevaluation.

The2009EvaluationPolicyhasputrenewedemphasis on evaluating all direct programexpendituresofgovernmentand,intheprocess,focusingontheissuesofprogramrelevanceandperformance—thatis,effectiveness,efficiency,andeconomy.Theintentisthatevaluationwillbe

9 RobertLahey.“TheCanadianM&ESystem:LessonsLearnedfrom30YearsofDevelopment” (Washington,DC: IEGECDWorkingPaperNo.23,November2010).

Over the past two decades, the World Bank and other inter-national financial institu-tions, and more recently some prominent UN agencies, have devoted resources to building evaluation capacity in selected countries.


betterused,bybothdepartmentsandthecentralagency,toaddresshigher-levelissues, and,ulti-mately,tobetterintegrateprogrameffectivenessinformationintodecision-makingprocesses,especiallyexpenditure-managementdecisions.

d) Columbia caseColombia’snationalM&Esystem(SINERGIA)isoneofLatinAmerica’smorevisibleeffortstoinstitutionalizeM&Eatthegovernmentlevel.

Establishedin1994,followingthemandatesoftheNationalConstitution,SINERGIA,10thenational system for evaluation of management and results,hasendurednumerousperiodsofprogress,stagnation,andsetbacksbecauseoftheinstitutional,political,andfiscalchallengesfacedbythecountryovertime.Nonetheless,SINERGIAhasattainedasignificantlevelofconsolidationandcustomizationandisoftenreferredtoasagoodpracticesystembymulti-lateralorganizations,donoragencies,andaca-demicanalysts.

Colombia’sM&Esystemsqualifyasgoodprac-ticebecause:

• ThequalityoftheirM&Eworkisgenerallyreliable.

• Themonitoringinformationandevaluationfindingstheyproducereceiveintensiveuse.

• Limitationsofrelyingonlaws,decrees,andregulations.

Finally,itappearsthatmanyfactors,suchasthestrengthofleadershipforthereform,theresourcesprovided to support it, the estab-lishmentofroutinerulesandprocedures,and

incentivesfortheutilizationofM&Einforma-tion,arealsocrucialtoasystem’ssuccess.

The new global realityOverthepasttwodecades,theWorldBankandotherinternationalfinancialinstitutions,andmorerecentlysomeprominentUNagencies,havedevotedresourcestobuildingevaluationcapacityinselectedcountries.However,inaworldwhereresourcesaretighterinallglobalsituations,thecontextfornationalevaluationcapacitydevelop-menthaschangedforUNagenciesandpartnercountriesalike.ThenewglobalrealityisthatUNagencies,bothontheirownandunderthecoor-dinatingbodyoftheUnitedNationsEvaluationGroup,willneedtobequiteselectiveinsearchingforthemostcost-effectiveapproachtosupport-ingnationalevaluationcapacitydevelopment.

AlongthelinesoftherecentBusanPartnership,11 civilsocietyorganizations(CSOs)canandshouldplayacentralroleinadvocatingfortransparencyintheallocationandexpenditureofthepublicbudget,accountabilityfortheimplementationofpublicpolicies,andthestrengtheningofthedemandfor theuseofevaluation to informevidence-basedpolicymaking,andthecapaci-tiesofqualifiedevaluatorstoproducecredibleandusefulevaluationsbasedonnationalandinternationalevaluationstandards.

InCôted’Ivoire,manyactionsandactivitiesarebeingperformedtofosternationalevaluationcapacitydevelopmentatthenationallevel.Theseincludethefollowing.

10 KeithMackayandothers.“ADiagnosisofColombia’sNationalM&ESystem,SINERGIA”(Washington,DC:IEGECDWorkingPaperNo.17,February2007).11 BUSANPartnershipforEffectiveDevelopmentCooperationFourthHigh-LevelForumonAidEffectiveness,BUSAN,RepublicofKorea,29November–1December2011.


a) The Ivorian M&E Network (RISE)RISEhasgainedofficialgovernmentrecogni-tionandispresentatallthemajormeetingsorganizedbytheAfricanEvaluationAssociation(AfrEA).

Oneofthemostrecentoutcomesisthepre-qualificationofRISEforthenationalassociationcompetitionorganizedduringthesixthconfer-enceoftheAfricanEvaluationAssociationinAccrainJanuary2012.

Mr.SamuelKouakou,onbehalfofRISE,par-ticipatedinthe10thBiennialConferenceoftheEuropeanEvaluationSociety(EES),whichwasheldinFinlandfrom1to5October2012.ThisactivitysignifiedtheinvolvementofRISEinmajorinternationalmeetingsonevaluation.

Manyinterviewshavebeenmadeavailableinnationalnewspapersandradiobroadcastsadvo-catingfordecisionmakersandthestrengtheningofevaluatorcapacities.

Finally,acasestudypresentedbyMr.SamuelKouakou,onbehalfofRISE,hasbeenselectedforinclusionintheInternationalEvalPartnersForum,organizedon3-6December2012inChiangMai,Thailand.Theaimwastostrengthentheevaluationcapacitiesofcivilsocietyorgani-zationsingeneral,andvoluntaryorganizationsforprofessionalevaluationinparticular.

Thenextstepswillbe:

• IvorianEvaluationDayswillbeorganized.

• Atasubregionallevel,experiencewillbesharedbetweenthevoluntaryorganizationsforprofessionalevaluationoftheEconomic

CommunityofWestAfricanStates.ThefoundationforthissharingaslaidduringthesixthconferenceoftheAfrEAinAccra.

b) The national institutions• Organizationofaworkshoptostrengthenthe

technicalcapacityofsectoralministriesandleadersofdepartmentsinchargeofplanningandstatistics,operationalfocalpointsintheimplementationoftheNationalDevelopmentProgram(NDP2012–15)andtothemethodsandforesighttools,strategicplanning,pro-gramming,budgeting,andM&E(PPPBSEcycle);

• The implementationof theM&EsystemofthetheNationalDevelopmentProgram(NDP2012–15)isstillinprocess;

• ThedesignoftheM&EsystemoftheNationalAgriculturalInvestmentProgram(PNIA2012–15)by theMinistryofAgriculturethroughtheDirectorateofEvaluationandProjects(DEP)isstillinprocess;

c) The International levelAsamemberoftheEvalPartnersTaskForceontheenablingenvironment,Iamdeeplyinvolvedintheimplementationofactivitiestoadvocateandpromoteevaluationandevidence-basedpolicymakingattheinternational,regional,national,andlocallevels,andindesignating2015astheInternationalYearofEvaluation(EvalYear).

Manyothercapacity-buildingactivitieswillbe conducted by the European EvaluationSociety(EES)WorkingGrouponSustainableDevelopment, as well the InternationalDevelopmentEvaluationAssociation(IDEAS)


WorkingGrouponCertificationforDevelopmentEvaluators,towhichIbelong.

Urgent issues in national evaluation capacity developmenta) The issues of certification of evaluatorsTheprofessionoftheevaluatorisyoung.Thereisnoprofessionalorderofevaluators.Therearenationalevaluationassociationsinseveralcoun-tries,generallyindustrializedcountries.

Therearemanyincompetentevaluators, justastherearemanyseasonedprofessionals.Thismakesitdifficultforthosewhosetouttohireanevaluatortoknowwhoisqualifiedtodothejobandwhoisnot.

Thefieldwouldbenefitfromevaluatorcom-petenciesinfourprimaryways:(a)improvedtraining,(b)enhancedreflectivepractice,(c)theadvancementofresearchonevaluation,and(d)thepotentialforcontinuedprofessionalizationofthefield.

ThenewlyaccreditedCLEARcenters in theLatinAmerica,Africa,andAsiaregionsthatareplanningtoofferuniversitycertificationsinthefuturewillassuredlyhelpaddresstheseissues.

b) The issues of enabling environmentWemustensurethat:

• Public administration is committed totransparencyandmanagingforresultsandaccountabilitythroughresults-basedpublicbudgetingandevidence-basedpolicymaking.

• Thereislegislationand/orpolicies,includingpublicbudget,toinstitutionalizeevaluationsystems.

• Therearenationalorganizationsforevalu-ationprofessionals.

Key elements for successful national evaluation capacity developmentInthepast,evaluationcapacitydevelopmentmainlyfocusedonstrengtheningindividualcapacities.However, it isclear thatcapacitydevelopmentshouldbebasedona systemicapproachthattakesintoaccountthreemajorcomponents:theindividual,institutional,and,externalenablingenvironments.12

Successfulnationalevaluationcapacitydevelop-mentmustbebasedoncountry-ledevaluationsys-tems.Therefore,internationalorganizationsanddonorsshouldalwaysmakesuretheirownevalua-tionfunctionstrengthensratherthanunderminesnationalevaluationsystemsandcapacities.

Inthepast15years,thepotentialofmajornationalandregionalvoluntaryorganizationsofprofes-sionalevaluatorsgrew,astheyroseinnumberfrom1inthe1981tomorethan135in2013.13

Factorsrelatedtothecomplementarylearn-ingstrategiesthatreinforceuseofthetraininginclude:

• Learningbydoingandmentoring:Itcanbeaneffectivefollow-upofthetraining.

• Self-education:Thiscanbeaneffectivestrat-egyforevaluationcapacitybuilding.

12 UNICEF. “Evidence for Children: Developing NationalCapacitiesforCountry-LedEvaluationSystems,AConceptualFramework.”(NewYork:UNICEF,July2010).13 J.RughandM.Segone,eds.Voluntary Organizations for Professional Evaluation (VOPEs): Learning from Africa, Americas, Asia, Australia, Europe, and Middle East (EvalPartners, UNICEF, IOCE, 2013), pp. 14–15.


• Learningbysharing:Thisisapowerfulwayofdeepeningevaluationknowledgeandcanbeconsideredoneofthebestpossibleoutcomesofevaluationtraining—thatis,peopleusingtheirnewlearningtoteachothers.

Actions that would strengthen evaluation capacity developmentAcountrystrategyfornationalevaluationcapac-itydevelopmentshouldstrengthentheenablingenvironment bysupporting:

• Thecommitmentofpublicadministrationtoaccountability,transparency,andmanagingforresults, includingstrengtheningresults-basedpublicbudgetingandevidence-basedpolicy-making.

• Legislationand/orpoliciestoinstitutional-izenationalevaluationsystemsshouldbedesigned,adopted,andimplemented.

• Nationalevaluationorganizationsshouldbesupportedtoenablethemtofosternationaldemandforandsupplyofevaluation,includ-ingbysettingnationalevaluationstandardsandnorms.

Attheinstitutionallevel,astrategyforevalu-ationcapacitydevelopmentshouldstrengthenthecorporateevaluation culture throughinsti-tutionalcommitmenttolearningfromevidenceproducedbyevaluationsystems.Itshouldalsosupportevidence-baseddecisionsanddemandforaccountabilityandestablishaprotective envi-ronment thatremovesrepercussionstocareers.

Finally,parliamentsandcivilsocietyorganiza-tionscanalsoplayamajorroleinadvocatingfortheuseofevidenceinpolicyimplementation.

Thesupportofmassmediacanalsomakeevi-denceavailable tocitizens,andcitizenscandemandthatpolicymakersmakemoreuseofit.

Agoodinitiativethatwillstrengthenevalu-ationcapacitydevelopment is theorganiza-tionParliamentariansForumonDevelopmentEvaluation in SouthAsia.14 The forumwasinitiatedbyasmallgroupofparliamentariansaimingtoestablishnationalevaluationpoliciesintheSouthAsianAssociationforRegionalCooperationregiontoensureaideffectiveness,achievementofresults,andsustainabilityofdevelopment.

14 http://www.mymande.org/evalyear/parliamentarians_forum

Mr. Samuel Kouakou is Deputy Director of Evaluation and Projects Sustainability in the Directorate of Evaluation and Projects of the Ministry of Agriculture, Côte d’Ivoire.

He has extensive experience in evalua-tion in public and private service at the national and international levels. He is an active member of the African Evaluation Association, International Development Evaluation Association, European Evaluation Society, and the Ivorian M&E Network (RISE).He holds a diploma in international environmental law from the United Nations Institute for Training and Research (UNITAR), a master’s degree in rural engineering, as well as several post-graduate certificates in the areas of integrated water resources management, gender mainstreaming, and results-based M&E.

Mr. Kouakou is currently e-learning visit-ing lecturer at the International Institute for Water and Environmental Engineering . He earned his e-learning tutor certifica-tion from the Association of Universities of the Francophonie.


F. Kelemework Fasika, European External Action Service-Addis Ababa

since the initiation ofPovertyReductionStrategyPapers(PRSPs),povertyreductionhasbecomethemainfocusofthenationalpoliciesandstrategiesofmostdevelopingcountries.Moreover,thisfocushasbeensharedbyregionalandglobaldevelopmentplayersforquitesometime.Althoughmuchhaschangedovertime,povertyandinequalityremainchallenges,andtheneedforstrongandsustainablegrowthisstillhighonthedevelopmentagenda.

Lackofadequatecapacityisconsideredafactorholdingbackprogressintheattainmentofthegoalsofnationalpovertyreductionstrategies,aswellasglobalcommitments,includingtheMillenniumDevelopmentGoals(MDGs).Theneedfordevelopmenteffectivenesshadbeenpartoftheglobaldevelopmentdiscourse.Thisledtothefor-mulationoftheParisDeclarationonAidEffectivenessin2005,andtheagendawasreviewedandfurtherstrengthenedthroughtheAccraAgendaforActionandtheBussanPartnershipAgreement.

Developingevaluationcapacityisacornerstoneinbuildingaholisticcapacitytodesignandimplementeffectivepoliciesandstrategies.Itprovidesinsightonthecur-rentsituation,helpsidentifyneededimprovements,andgenerateslessonstoenablefutureprogress.Basedonthisunderstanding,theobjectivesofdevelopingevaluationcapacityentailpromotingefficiencyandeffectivenessofdevelopmentinterventions;enhancingimplementationcapacitytorespondtothegrowingneedsforservicedelivery,jobcreation,andthelike;andinformingdialogue(bothonpolicyandonevaluationcapacitydevelopment).Throughevaluation,governmentscanholdtheircivilservicesaccountableforeffectiveuseofdevelopmentresources,aswellasfortheirperformance.Evaluationcanalsohelpgovernmentstobemoreaccountabletothecitizensandgainlegitimacy.

Forquitesometime,whateverthelevelofcapacity,everydevelopingcountryhadasystemtomonitorandevaluateitsdevelopmentefforts.However,thefocusofsuchsystemshadbeenoninputsandprocessesratherthanonoutcomesandimpact.Apartfrommeasuringinputsandprocesses,theeffectivenessofanynationalpolicyandstrat-egydependsonlearninglessonsfromwhatworks,whatdoesn’t,andwhy.Collecting

The Importance of Strength-ening National Evaluation

Systems - from Ethiopia


evidenceandmeasuringresultsthroughevalu-ationareessentialinprovidinginsightandnewperspectivestoinformpoliciesandstrategies.Moreover,astheaideffectivenessagendacameintobeing,progressbroughtaboutbydevelop-mentinterventionswasmeasuredintermsofresultsandimpacts.Theroleofmonitoringandevaluation(M&E)asatooltoinformthedesignofnationalplansandstrategies,assessingresultsaswellasmeasuringimpactandchanges,hasincreasedtremendously.Measuringdevelopmenteffectiveness, learninglessons,andadjustingfutureactionbasedonevidencegeneratedhasbecomeincreasinglyimportant.Nationalevalu-ationsystemsplayakeyroleinthisregard.

Basedonthediscussionabove,theneedforstrength-eningnationalevaluationsystemsemanates,first:domestically,fromgovernmentswithaninterestinbecomingmoreefficientinoptimizingresourcesandbecomingmoreeffectiveinachievingdevelop-menttargets,aswellasdischargingtheaccount-abilityfunctiontotheircitizensand,second,frominternationalcommitmentsandprinciples,includ-ingtheMDGs;ParisAidEffectivenessAgenda;AccraAgendaforAction;and,mostrecently,theBussanPartnershipAgreement.

Tobecomemoreefficientandeffective,aswellastodischargetheiraccountabilityfunctiontocitizens,governments,civilsocietyorganizations,theprivatesector,andotherrelevantstakeholdersneedtoworktogethertodeveloplocalevaluation

capacitiesinallsectorsandatalladministrativelevels.EvidencegatheredthroughM&Esystemsprimarilyhelpsgovernmenttodemonstratetheeffectivenessofdevelopmentpoliciesandstrate-gies,mobilizeresources,useevaluationresultstocampaignforelection,andobtainlegitimacy.Evaluationcanalsohelpgovernmentstoidentifytheirstrengthsandweakness,aswellaslearnlessonstoenhanceperformance.

Thesecondreasonthatstrongerevaluationcapac-ityisneededemanatesfromtheinterestofdevel-opmentpartners.Theyneedtobeabletotracktheeffectivenessandefficiencyofdevelopmentinterventionsandensureaccountabilityintheuseofdevelopmentresourcesprovidedtorecipientcountriesbasedonproject,program,orbudget-supportinterventions.Internationalcommitmentsandprinciples,includingtheMDGs,ParisAidEffectivenessAgenda,AccraAgendaforAction,andBussanPartnershipAgreement,whichareendorsedbybothdonorandrecipientcountries,aredesignedtopromotedevelopmenteffectivenessandmutualaccountability.Theseglobalcommit-mentsfocusondevelopmentresultsandencouragecountriesthatreceivedevelopmentassistancetocreatetransparentresultsframeworkstomonitorprogress,evaluateimpact,andhighlightstrategicissuesthatrequirepolicydecisionsandbecomemoreeffectiveandefficientastheyworktowardmeetingtheirdevelopmentgoals.DonorcountriesworktoenhanceM&Ecapacitybyprovidingtech-nicalknow-howandfinancialsupport.

ChallengesDeliveringonresultsandachievingfastereco-nomicgrowthandsocialdevelopmentacrossthesocialandeconomicsectors—atboththelocalandnationallevels—inadecentralizedsystemrequires

acarefullydesignedsetofpoliciesandstrategieswithstrongandharmonizedM&Esystems.Theeffectivenessofanynationalpolicyandstrategydependsonlearninglessonsfromwhatworks,


whatdoesn’t,andwhy.However,answeringsuchquestionsdependsoncreatingandmaintainingsoundandharmonizednationalM&Esystemsinvariousadministrativelevelsandsectoralareas,whichposesbothtechnicalandadministrativechallenges.Thesemustbeaddressedbasedonstudyandunderstandingofsuccessfulexperiencesfromcountrieswithsimilarcontexts.

Therelevanceofdevelopmentpoliciesandstrat-egiesfadeswithtimeandwithchangesinthecountrycontext.Thisrelevance,aswellaseffi-ciencyandeffectiveness,dependsonperiodicassessment,review,andamendment.Thatiswhyevaluationisvitaltoourabilitytoadjustpoliciesasweprogress.M&Eis,therefore,anessentialelementindevelopment.Policiesandprogramshaveuncertainties.CriticalM&Eisneededtoenabletimelyadjustmenttoenhancetheirrel-evance,efficiency,effectiveness,impact,andsus-tainability.Results-guidedlearningisessentialfordevelopingcountriessuchasEthiopia.Tofacilitatethis,governmentalandnongovern-mentalevaluationservicesneedtoworktogether.

SincetheadoptionofthePoverty-ReductionStrategyPapers,resultsframeworksandM&Esystemshavebecomeroutineannexes.Itistruethat almost all the organizationsprovidingservicesinEthiopiahavesomekindofbuilt-inevaluationaspartoftheirprocessofplan-ningand/oroperations.Butthisimpliestwoimportantthings.Ontheonehand,itindicatesthatevaluationiscommonlyused.Ontheotherhand,itleadstoageneralunderstandingthatevaluationiscommon,anditssignificancehasknowinglyorunknowinglybeeneroded,andinsomeorganizationsithasbeennothingmorethanacliché.Suchpracticesriskovershadowingtherealbenefitsofevaluation.

However,variouschallengesarehamperingtheabilitytoprovidetheaccurateevidencerequiredtofine-tunepoliciesandmakeevidence-baseddecisions.

Themainchallengesare:

• Properawarenessandunderstandingofthebenefitsofevaluation:althoughthereisanincreasingtrendofawareness,andM&Eunitsarebecomingcommonineveryinstitu-tion,thesignificanceofevaluationhasbeenknowinglyorunknowinglyoverlooked,andthisovershadowstherealbenefit.

• Technical know-how on evaluation: aknowledgegapinthetechnicalapplicationofevaluationprinciples, techniques,andtools,aswellasthedesignofappropriateM&Esystems.

• HarmonizedM&Eprocesses,guidelines,andstructureswithdetailsofdatacollectionandanalysismethods,dataqualityassurance,andfeedbackmechanismsfromthelocaltothenationallevel.

• Strong statistical systems and technolo-giesthatdeliveronthedataneedsofM&Esystems.

• Coordinationandharmonizationamongevaluationcapacitydevelopmentapproaches.

• ExistenceofdifferenttypesofM&Esystems,whichmakesdataaggregationdifficult.

• Indicator development: limited focusonimpact;morefocusonprocessesandoutputs.


role of Professional Associations in Evaluation Capacity Development

TheEthiopianEvaluationAssociation(EEvA)hasbeenestablishedandlegallyregisteredinEthiopiasince2008.Itsvisionistobecomeacenterofexcellenceincompetencyandstand-ardsofM&EinEthiopiaandinAfrica.Itsmis-sionistoengageandenableM&Eprofessionalsinthecountrytocontributetoevidence-baseddecisionmakingandpositivelyinfluencethesocioeconomic development of the countrythroughknowledgegeneration,awarenesscrea-tion,capacitybuilding,advocacy,anddemon-stratingmodelserviceinevaluation.MembersoftheassociationareM&Eprofessionalsworkinginthegovernment,nongovernmentalorganiza-tions,academia,andtheprivatesectorinthecountry.

Aspartof itsmissiontodevelopevaluationcapacityinthecountry,theassociationiscur-rentlyprovidingtraininginM&Etoprofes-sionalsworkinginthegovernmentalandnon-governmentalsectors.Theassociationisalsoworkingtowardpromotingprofessional,qualityevaluationpracticeinEthiopiabydevelopingevaluationstandardstopositivelycontributetothedevelopmentobjectivesofthecountry.Toachievethis,itcreatesnetworkswithevaluation

professionals,disseminatesevaluationinforma-tion,andpartnerswithlike-mindedlocalandglobalorganizations,amongotheractivities.

EEvAconsidersitsroleinstrengtheningnationalevaluationcapacityasincludingthefollowingtasks:

• CreateawarenessandsensitizationonthebenefitsofM&E.

• Providetrainingandfacilitateexperience-sharingamongM&Eprofessionalsingovern-ment,nongovernmentalorganizations,andtheprivatesector.

• SupportdevelopmentofM&Esystems.

• Disseminateevaluationinformationtostake-holdersthroughpublications,brochures,andelectronicmedia.

• Conduct project, program, and policyevaluation.

• Developevaluationstandards.

TheWayForwardLeadership & ownership of evaluation capacity developmentThemostimportantaspectofevaluationcapacitydevelopmentistheissueofownershipandlead-ershiptocreatesustainablecapacity.Thevisionaswellasthemanagementoftheprocessshouldbeownedandledbythecountry;developmentpartnerscanplayasupportrole.Ownersshould

leadtheevaluationcapacitydevelopmentneedidentification,thedesignofinterventions,andmanagementoftheprocess.Partners,however,cancreateincentivesandresources.Nevertheless,themainincentivetodevelopevaluationcapacityshouldcomefromtheinsideintheformof,forexample,theneedtoimproveperformanceandaccountabilitymechanisms.


Evaluationcapacitydevelopmentcanonlybesustainablewhentheweaknessesandgapsintheexistingcapacityarewellunderstoodandthegapsthatneedtobefilledareidentified—mostimportantly,bytheowner.Thiscanbecontem-platedthroughassessmentofexistingcapacity.Suchastartingpointprovidestremendousroomandpowerforchangeandcanserveasadriverfortheprocess.Moreover,understandingthecurrentsituationisabasicpreconditiontothedevelopmentofavisionforimprovedcapacityinevaluation.

Havingaclearvisionandunderstandingtheneedforimprovementprovidestheconfidencetochangeandoffersaclearroadmaptonavigatetheprocess.Whensuchavisionispresentatthepoliticallevel,itprovidesenormouspoliticalwillandtherequiredresourcestosupporttheprocess.

Asinmostdevelopingcountries, ithasbeenobservedinEthiopiathatinadequateattentionisgiventoevaluation.Inaddition,otherchal-lengeshavebeennoted,includingunclearrolesandresponsibilitiesamongevaluationunitsatdifferentlevels,lackofcommitment,irrelevantandlow-qualityinformation,pooraccountabil-ityforfailures,lackofintegrationofactorsintheevaluationsystematdifferentlevels,problemsofmainstreaminglessonsdrawnfromevaluationresults,narrowingofthescopeofevaluation,poorevaluationcapacity,andlackofintegrationofdifferentcomponentsofthesystem.

Strategiestobuildevaluationcapacity:

• CreatepoliticalchampionsforM&E.

• Promoteengagementofcivilsocietyorgani-zations,academia,andresearchinstitutesinevaluationcapacitydevelopment.

• LinkthebenefitsofM&Etoeffectivepolicymak-ing,enhancinggovernmentefficiencyinopti-mizingresources,anddeliveringmoreresults.

• Strengthendonorgovernmentpartnershipanddialogue.

• Enhancetheimplementationofjointdonor-governmentaideffectivenesscommitmentsandimplementationmechanisms.

Ingeneral,keyelementsforsuccessfulnationalevaluationcapacitydevelopmentincludethefollowingsteps:

• Strengthentrainingandeducationinevaluation.

• Sensitizeandcreateawarenessamongpolicymakers,implementers,andthecitizenry.

• ProvidetechnicalsupportfordevelopmentofM&Esystems.

• Enhancedonor-governmentcollaborationanddialogue.

• Organizeeventstocreatepublicawarenessofevaluation,suchasanevaluationweekandalearningeventtodiscuss,debate,andshareles-sonsandexperience,aswellastoidentifyandaddresschallengesinpromotingevaluation.

Actionsthatwouldstrengthennationalevalua-tioncapacitydevelopmentinclude:

• Buildthecapacityofindividualevaluators.

• Strengthenevaluationsystemsandstructures.

• Developharmonizedevaluationframeworks,guidelines,andmanuals.


Dr. Ian Goldman, Head, Evaluation and Research in the Department of Performance M&E, South African Presidency. [email protected]

Backgroundduring the 2000s therewasagrow-inginterestinmonitoringandevaluation(M&E)inSouthAfricaandtheM&Eroleinthepresidencybegantostrengthen.In2005thecabinetapprovedaplanforthedevelopmentofagovernment-wideM&Esystem.Itwasenvisagedasasystem of systems,inwhicheachdepartmentwouldhaveafunctionalmonitoringsystem,andthenecessaryinformationcouldbeextracted.

In2007apolicyframeworkwaspublishedtoguidethegovernment-wideM&Esys-tem.Itincludedtheneedforframeworksforprogramperformanceinformation,thequalityofstatisticaldata,andevaluation,andintheprocesssoughttostrengthenthelinkagesamongthepresidency,thetreasury,andtheNationalStatisticsAgency.Policyframeworksweredevelopedforthefirsttwoelementsin2007,butnotforevaluation.

Thegovernmentthatcametopowerfollowingthe2009electionsfacedanumberofchallenges,whichincluded1)persistentpovertyandinequality,2)widespreadservicedeliveryprotestsatthemunicipallevel,and3)lossofsomepoliticalsupportintheelections.Thesechallengesresultedingreaterwillingnessoftherulingpartyandthegovernmenttobefrankaboutthepoorqualityofpublicservices,corruption,andothergovernanceproblems,aswellasapoliticalconsensustoimprovegovernmentperformance,includingagreaterfocusonM&E.AMinistryofPerformanceM&Ewascreatedinthepresidencyin2009,andaDepartmentofPerformanceM&E(DPME)inJanuary2010.Inaddition,anadvisorybody,theNationalPlanningCommission(NPC),wasestablishedinthepresidencytofocusonalong-term2030plan.

1 ThefirstpartofthepaperdrawsheavilyfromGoldmanandothers2012.

Developing a National Evaluation System in

South Africa1


DPME started by building on the UnitedKingdommodel establishedbyTonyBlair’sadministration,focusingonalimitednumberofpriorityoutcomes,developingministerialtargets for these,plans todeliver them,andthenworkingwith relevantdepartments tounblockdeliveryoftheseplans.Thenextareaofwork—developedbasedontheexperienceofCanada—wastheassessmentofmanagementperformanceofdepartments,theManagement

Performance Assessment Tool (MPAT). In2011 the focus shifted to evaluation. VisitsweremadetoMexico,Colombia,theUnitedStates,andAustralia.TheexperiencesofMexicoandColombiawereusedtodevelopaNationalEvaluationPolicyFramework(NEPF),whichwasapprovedbycabinetinNovember2011,andtheexperienceofAustraliapre-1996wasusedtodevelopthemodelforaNationalEvaluationPlan.

TheApproachfortheNationalEvaluationSystemHistorically,workaroundevaluationinSouthAfricawasdevelopedinthenongovernmen-talsectorinresponsetodonorrequirements(Podems, Jacob, andGoldman 2013). Somedepartments starteddevelopingevaluationsinthe2000s,butworkwassporadic,withnonationalsystemorstandards.Hence,althoughmostdepartmentshaveM&Eunits,inpracticetheworkthathasbeenundertakenisprimarilymonitoring.Accordingtothe2013ManagementPerformanceAssessmentsjustreleased,only17percentofdepartmentsareundertakingorplanningtoundertakeevaluations.Asaresult,departmentsaremissingoutontheopportunitytoimprovetheeffectiveness,efficiency,sustaina-bility,andimpactoftheirpoliciesandprograms.

ThedesignofthesystemasenvisagedintheNEPFisseekingto:

• Improveprogramperformance.

• Strengthenaccountability,

• Improvepolicyanddecisionmaking.

• Improveknowledgeofwhatworksandwhatdoesn’t.

Astrategicapproachhasbeentaken,focusingonimportantpolicies,program,andplans,andtheselectedevaluationsareembeddedinaNationalEvaluationPlan,with15evaluationsselectedeachyearduring2013/14.Thefocushasbeenonutilization,andallevaluationswillbemadepublicallyavailableunlessconfidential,andtheymusthaveanimprovementplan,whichisthenmonitored.Theapproachemphasizeslearningratherthanapunitiveapproach.Thegoalistobuildevaluationintothecultureofdepart-ments,notpromoteresistanceandmaliciouscompliance.

Sixtypesofevaluationsareenvisaged,whilespe-cificevaluationsmaycombinetheseapproaches:

• Diagnostic—Identifies the root cause ofproblemsandtheoptionsthatcouldbecon-sideredforaddressingthem.

• Design—AshortevaluationofthedesignofprogrambyM&Eunitswithindepartments


toensuredesignsarerobust,ideallybeforeimplementationstarts.

• Implementation—Ref lects on the pro-gressofaninterventionandhowitcanbestrengthened.

• Impact—Identifiestheimpactandattribu-tionofinterventionsandhowtheycanbestrengthened.

• Economic—Looksatthecost-effectivenessorcost-benefitofinterventions.

• Evaluationsynthesis—Drawsoutlessonsacrossanumberofevaluations.

Evaluationsareimplementedasapartnershipbetween the department(s) concerned andDPME,andDPMEpartiallyfundstheevalua-tions(withanaverageofUS$60,000perevalua-tion).AnEvaluationandResearchUnithasbeenestablishedinDPMEtodrivethesystemandprovidetechnicalsupport,backedbyacross-government Evaluation TechnicalWorkingGroup.Theunithad15staffasat1September2013,ledbyadeputydirectorgeneral.Guidelinesarebeingdeveloped,aswellasstandardsforeval-uationandcompetenciesforprogrammanagers,M&Estaff,andevaluators.TrainingcoursesstartedinSeptember2012.

TheCurrentSituationWithEvaluationsThefirstevaluationtopilotthesystemwasonearlychildhooddevelopment.ThisevaluationwascompletedinJune2012andtherecom-mendationsintheimprovementplanarebeingimplemented.ThefirstNationalEvaluationPlanwasapprovedbythecabinetinJune2012,andworkisnowprogressingonsevenoftheeightevaluations includedintheplan,withthefinalreportsapprovedfor twoof these,oneonthereceptionyearofschooling(gradeR),andoneonaBusinessProcessOutsourcingIncentive Scheme. Fifteen evaluations for2013/14areunderwayorintheprocurementphase,andthe15for2014/15havejustbeenselected.Theseareprimarilyevaluationofpro-grams.Thefirstpolicyevaluationisscheduledfor2014/15,alongwithadesignevaluationofapolicy.

Asimilarprocessisunderwayattheprovin-ciallevel,andDPMEiscurrentlyworkingwith

threeprovincesonprovincialevaluationplans.Threedepartmentshavenowdevelopedpro-vincialevaluationplansthatcombineinternalevaluationswiththoseproposedfortheNationalEvaluationPlan.

Akeyelementinbuildingthesystemtodatehasbeendevelopingevaluationstandards(basedontheOrganisationforEconomicCo-operationandDevelopmentAssistanceCommitteestand-ards),guidelines,andtemplatestoensurethatevaluationsareofsufficientqualityandcredibil-ity.Thirteenguidelinesandtemplateshavebeenproduced,andallareavailableontheDPMEwebsite(www.ThePresidency-dpme.gov.za).

Inadditiontocommissioningnewevaluations,anaudithasbeendonetoidentifyexistingeval-uationsundertakenbetween2006and2011.Eighty-threeevaluationshavebeenidentified,andaqualityassessmenttoolwasdeveloped


basedontheevaluationstandards.Basedonthistool,71oftheevaluations“passed”theminimumqualitystandards.Thedocumentsforthese71

evaluationswillbemadeavailableinanevalu-ationrepository,whichisavailabletothepublicfromtheDPMEwebsite.

BuildingEvaluationCapacityAsquotedearlier,only17percentofdepartmentsareundertakingorplanningtoundertakeevalu-ations,andthislowparticipationratemakesexperiencearoundevaluationquite limited.Thereisalsolimitedtrainingaboutevaluation(Podems,Jacob,andGoldman2013).HencethedevelopmentoftheNationalEvaluationStrategy(NES)iseffectivelystartingwithacleanslate,althoughithasbeenabletousetheexperienceof a fewdepartments thathavebeencarry-ingoutevaluation,notablythePublicServiceCommissionandtheDepartmentsofSocialDevelopmentandBasicEducation.

Clearlythereisamajorlackofcapacity,andwhiletheNationalEvaluationPlanandprovin-cialanddepartmentalevaluationplansarecreat-ingdemand,thereisanissueofhavingsufficientsupplycapacitytobeabletodelivertheseevalua-tions,andtodeliverthemwithsufficientquality.

Asetofevaluationcompetenceshasbeendevel-opedforprogrammanagers,M&Estaffingov-ernment,andevaluators(alsoavailableontheDPMEwebsite),andthesehavebeenusedtodevelopasuiteoftrainingcourses.Thecom-petencescover:

1 Overarching Considerations

1.1Contextualknowledgeandunderstanding

1.2 Ethicalconduct

1.3 Interpersonalskills

2 Evaluation Leadership

3 Evaluation Craft

3.1Evaluativedisciplineandpractice

3.2 Researchpractice

4 Implementation of Evaluation

4.1 Evaluationplanning

4.2 Managingevaluation

4.3 Reportwritingandcommunication

4.4 Improvement

Table 1 shows an example of the first twocompetencies.


Table 1: Extract from South Africa’s evaluation competencesDomains/ Descriptors Program Manager M&E Advisor Evaluator

1.1Contextualknowledgeandunderstanding

CK1—Sector-specific knowledge:KnowledgeofrelevantsectorsinSouthAfrica(forexample,content,institutions,people,andpolitics)andcanappropriatelyrelatetheevaluationtothecur-rentpolitical,policy,andgovernanceenvi-ronmentsandissues

Canexplainrele-vantsectorinSouthAfrica(forexample,content,institu-tions,peopleandpolitics)andcanappropriatelyrelatetheevaluationtothecurrentpolitical,policy,andgovern-anceenvironmentsandissues

CanlearnaboutrelevantsectorsinSouthAfrica(forexample,content,institutions,people,andpolitics)andcanappropriatelyrelatetheevalua-tiontothecurrentpolitical,policy,andgovernanceenvi-ronmentsandissues

CanexplainrelevantsectorinSouthAfrica(forexample,content,institutions,people,andpolitics)andcanappropriatelyrelatetheevaluationtothecurrentpolitical,policy,andgovern-anceenvironmentsandissues

CK4 - Intervention knowledge: Understandsthespecificinterventionthatisthefocusoftheevaluationandhowandwhyitwasdevel-opedandimplemented

Canguideoth-ersonthespecificinterventionthatisthefocusoftheevaluationandhowandwhyitwasdevelopedandimplemented

Canlearnaboutthespecificinterven-tionthatisthefocusoftheevaluationandhowandwhyitwasdevelopedandimplemented(degreeoffamiliarityneededmaydiffer)

Thecoursesareshort,aroundthreedays inlength,timedatthepointintheevaluationcycleatwhichtheoutputsofthecoursearerelevant,whichgivesthemastronglearning-by-doingfocus.Thesuiteincludes:

• ManagingEvaluations:targetsprogramman-agers(threedays)andM&Estaff(twoxthreedays),andfocusesondevelopinggoodevalu-ationtermsofreference.Thiscoursewasbeendevelopedandrolledoutforevaluationsinthe2012/13and2013/14NationalEvaluationPlan,aswellasfortwoprovincesthathavedevelopedprovincialevaluationplans.

• DeepeningEvaluations:targetsbothpro-grammanagersandM&Estaff(threedays).Thistrainingfocusesoncriticallyinterpret-ingevaluationresults.Thecoursehasbeendesignedandwillbepilotedandrolledoutin2013.

• Methodology:basedonfiveguidelinesthatarebeingfinalizedonthedifferenttypesofevaluations,andfocusesonhelpingM&Estafftoselecttheappropriatetypeofevalua-tiondesign.Thiswillbedesignedandpilotedin2013.


• PlanningProgramsandDesignEvaluation (seebelow)

• Evidence-BasedPolicyMaking:focusingontopmanagers(seebelow)

Oneoftheareasthathaveprovedtobeveryweakisthedesignofprograms,whichisalimitingfac-torinconductingevaluations,butalsoinitselfamajorlimitationonprogramperformance.AguidelinewasreleasedinJuly2013onPlanningImplementationPrograms,withastandardlog-frame,requirementsfordiagnosis,riskanalysis,andsoon.Thisislikelytohaveamajoreffectinimprovingprogramdesign.Arisingfromthis,

anotherguidelinewillbedevelopedonDesignEvaluation,wherebyM&Eofficialswillreviewthedesignofnewprogramstoseethattheyarerobustandlikelytoachievetheirintendedobjectives.Acoursetosupportthisplanningprocessanddesignevaluationwillbedevelopedandpilotedin2013.

Toassistwithsupplyofqualityevaluations,DPMEhascreatedapanelofevaluationserviceproviders,tobeusedfortheevaluationsintheNationalEvaluationPlan,butlikelytoemergeasapanelacrossgovernment.Thepanelhasbeenexpandedto42organizations,butwillneedtobeenlargedasittakesonthiswiderrole,andthecapacityoftheorganizationsenhanced.

BuildingDemandforEvaluationAmajorlimitingfactorfortheevaluationsystemishowM&Eisviewed.DPMEconductedasur-veyin2012tounderstandthisbetter(Goldman,Ntakumba,andJacob2013).Intermsofculture-basedbarriers,morethanhalfoftherespond-ents(54percent)indicatedthatproblemsarenottreatedasopportunitiesforlearningandimprovement.Thesurveyalsorevealedthatsen-iormanagementoftenfailstochampionM&E(45percent),M&EisregardedasthejoboftheM&Eunitratherthantheresponsibilityofallmanagers(44percent),thereisnotastrongM&Eculture(40percent),M&Eisseenaspolicingandcontrolling(39percent),andM&Eunitshavelittleinfluence.TheseallpointtothechallengeinusingM&Easastrategicfunctiontoinformpolicyanddecisionmaking.

In57percentofcases,M&Einformationhadlimitedornoinfluenceondecisionmaking.Nearlyhalf of the respondents (46percent)regardintegrationwithpolicydevelopmentas

eithernonexistentorverylimited.Justunderhalfofthedepartments(48percent)reportedthatintegrationofM&Ewithbudgetingislimited.Thislackofintegrationimpliesapoorenvi-ronmentforthedemandforanduseofM&Eevidence,sinceitislikelytobeviewedasastand-aloneactivity,detachedfromotherkeymanage-mentprocesses.

Thishasseveralimplicationsfortheevaluationsystem:

• Thereisapushtomakeitademand-drivenandlearning-focusedsystemtoovercomethecompliancementalityandmakeitlikelythatevaluationresultswillbeusedtoimproveperformance.

• Seniormanagersneedtobuyintothesystemandseeitasimportantfortheirownanddepartmentalperformance.


• Othersourcesofdemandneedtobestimu-lated,includingthecabinet(whichapprovestheNationalEvaluationPlanandreceivesreportsonevaluations)andparliament,tofacilitatetheiroversightfunction.

• Itisimportantthatevaluationsbemadepub-lictostimulatedemandfromthecitizenryandtofosteruseofevaluationresults.

With these implications inmind,DPMEhasdecidedtoconducttrainingforseniormanagers.Acase-basedthree-daycourseisbeingdevelopedfortopmanagers(directorsgeneral/permanentsec-retariesanddeputydirectorsgeneral/deputysecre-taries),whichisbeingpilotedinNovember2013.Tomakeitmoreinterestingtotheparticipants,thefocusisonevidence-basedpolicymakingandimplementation,andinternationalexperienceisbeingbroughtinwiththesupportfromtheCenterforLearningonEvaluationandResults(CLEAR).

Inaddition,intensiveworkisbeingundertakenwithparliament.DPMEactually reports totheStandingCommitteeonAppropriations.Workshopshavebeenrunforthecommittee,and study toursundertaken—to theUnitedStates andCanada in 2012, andKenya andUgandain2013—tobuildtheunderstandingofthecommitteearoundM&E.Thishasbeenverysuccessful,andDPMEhasnowdonepresenta-tionstothechairsofallportfoliocommittees,toparliamentaryresearchers,andtospecificportfoliocommittees,tostimulatetheiraware-nessaroundevaluationsandhowtheycanusetheresultstofacilitatetheiroversightrole.InJune,thefirstcaseswerepresentedtoportfoliocommittees,andthisledtorequeststodepart-mentstosubmitspecificevaluations.DPMEisnowplanninganextensiveprogramofcapacitydevelopmentforparliamenttoassistcommitteesinusingDPME’sM&Einformation.

TheRoadAheadThispaperhasoutlinedwhereSouthAfricaisintermsoftheNationalEvaluationStrategy.Keyissuesthatwillemergeinthenextfewmonthsincludehowdepartmentsactwhenevaluationresultsarechallenging,andtheimplicationsofthisfortheevaluationsystem.Thenextyearwillbeaninterestingone,as23evaluationsconclude.

Capacity in a number of departments willincrease,becausetheyareinvolvedinseveralshortcoursesandwillcompleteanumberofevaluations(forexample,ruraldevelopment,tradeandindustry,andhumansettlements).Thisexperiencewillcreateacadreofexperiencedevaluationmanagers.

Thereisanemergingchallengeinbeingableto supply sufficientnumbersofhigh-qualityevaluators.DPMEconductedtrainingaroundtheSouthAfricanM&EAssociationConferenceinSeptember2013,specificallytargetingexperi-encedevaluators.

Someoftheissuesthatemergedinclude:• Amongevaluatorsfromcompanies,there

areareasofweaknessinbasicresearchcom-petences,suchasusingliteraturereviewstogenerateanalyticalframeworkstoguidetheevaluation.

• Amonguniversities,thereareweaknessesinbeingabletowriteforapolicyaudience


andtogeneraterecommendationsthatarecloselylinkedtoimplementationrealities.

• Theneedtowriteshortreportsisachallenge.TheNationalEvaluationStrategyrequiresthatallfinalevaluationreportsusea1/3/25format—one-pagepolicysummaries,3-4pageexecutivesummaries,and25-pagemainreports—toensurereadabilityforbusyman-agersandpolicymakers.

• BearinginmindtheSouthAfricancontext,thereisashortageofexperiencedblackeval-uators.Aswellasbeingaweaknessinitself,this leadstoalackofcredibilityofsomeevaluators,withawarinessamongpolicymakersaboutusing(thesame)oldwhitemaleresearchersandevaluators.

AllofthispointstotheneedtoinfluencetertiarycoursesinM&Eandtotakenoteofthecompe-tencesneededaroundevaluation,andDPMEisindiscussionswithtertiaryinstitutionstothisend.

referencesDPME(DepartmentofPerformanceM&E).2011.

“NationalEvaluationPolicyFramework.”Pretoria,DepartmentofPerformanceM&E,availableatwww.thePresidency-dpme.gov.za.

———. 2012. “Evaluation CompetencyFramework for Government.” Pretoria,DepartmentofPerformanceM&E,avail-ableatwww.thePresidency-dpme.gov.za.

Goldman, I., S. Ntakumba, and C. Jacob.2013.“Reflectionson theSouthAfricanExperiencewithEvaluationandtheUseofEvaluativeEvidence toOrientPublic

Policy Formulation.” Paper for theUNDPEvaluationCapacityDevelopmentConference,SaoPaolo,September2013.

Goldman,I.,R.Engela,I.Akhalwaya,N.Gasa,B.Leon,H.Mohamed,andS.Phillips.2012.“EstablishingaNationalM&ESysteminSouthAfrica.”PREMnotesNo.21,SpecialEdition on Nuts and Bolts of M&E.Washington,DC,WorldBank.

Podems,D.,C.Jacob,andI.Goldman.2013.“Evaluator Competencies: The SouthAfricanExperience.”PapersubmittedtoCanadian Journal of Evaluation.

Dr. Ian Goldman started his career as an agronomist in Mexico in 1978-79, and has since worked in 18 countries in Africa, Europe, and Latin America, and with national, provincial, and local government, as well as in the NGO and small and medium-size enterprise sectors.

In October 2009, Dr. Ian Goldman became Team Leader of the Monitoring and Learning Facility of the Programme to Support Pro-Poor Policy Development (PSPPD) and in January 2011 he was appointed a deputy director general in the Department of Performance M&E (DPME) in the presidency, where he is responsible for evaluation and research. In this capacity he is leading the establishment of the National Evaluation System. Many elements of the system have been estab-lished, including a National Evaluation Policy Framework, guidelines and templates, evaluation standards, evaluation competencies, and 5 courses; 20 evaluations are under way.


David Rider Smith, UK Department for International Development’s Evaluation Adviser in Uganda and Acting Head of the Capacity and Quality Group in its Evaluation Department 1

ContextOver thepast twodecades, consider-ableeffortshavebeenmadetoestablishastrongandrobustbasisforassessingpublicspendinganditseffectsonthedevelopmentofUgandaanditscitizens.Tobetterunderstandthelinkages,patternsandconstraintstogrowthandchange,substantialresourceshavebeenspentonestablishinggoodtime-seriesandqualita-tivedataonkeysocio-economicindicators;publicaccounts;regularmonitoringofpublicpolicyinterventions;and,onpolicyrelevantresearch,analysisandevaluation.

Inordertounderstandwherethisinvestmentinmonitoring,evaluativeresearch2 andevaluationhashadthebiggestimpactonpublicpolicyandaccountability,itisnecessarytoexaminetherelationshipsbetweenpolicy,institutionsandindividualsinthepublicsphere.Evidencesuggeststhatonlywhentheenvironmentineachofthesespheresisconduciveandinterfacedhasthelinkagebetweenassessmentandpolicychangebeenproductive3.

1 DavidRiderSmithisanEvaluationAdviserwiththeUKDepartmentforInternationalDevelopment.Allviewshereinaresolelythoseoftheauthoranddonotrepresenttheviewsofhisemployer.2 Inthispaper,evaluativeresearchconductedinthepublicsector,i.e.ledbypublicorquasi-publicsectorinstitutions,referstoanalysisnotonlyoftrends,butalsoofcausesandpotentialpolicyresponses.TheseincludethePovertyStatusReports(PSRs)andrelatedanalyticalproducts.3 Thispaperbuildsononepresentedatthe2012AmericanEvaluationAssociationConferenceentitled‘politicaleconomyofevaluationinUganda’spublicsector’andthearticlebyRider Smith, Nuwamanya and Nabbumba Nayenga, 2010,Policies,InstitutionsandPersonalities:LessonsforUganda’sexperienceinMonitoringandEvaluationinFrom Policies to Results: Developing capacities for country monitoring and evaluation systems,UNICEF.

Patterns and Influences in the Supply and Demand

of Evaluation and Monitoring in Uganda’a Public Sector over

the Past Two Decades


Normative FrameworkRelevantliterature4pointstothecriticalpre-requisiteofavarietyofdemand-sideelementsintheuseofevaluationandevaluativeresearchinpublicpolicyandprogrammaticdecisionmaking.Theseelementscanbecategorizedintofourthatpertaintotheopennessofthepoliti-calsystemtoevidenceandargument;organi-zationalsystemsthathaveelementsofperfor-mancemeasurementandanalysisembeddedwithin; individual leadershipwhererelevantpolicymakershaveaninterest inanalyticalworkand/orknowledgeofrelevantsubjectmat-ter;andindividualevaluationfactorsrelatingtothetimingandfocusofevaluations.Whilethere isnoempiricalevidenceonthedepthorextent towhichthese factorsneedtobepresentoractive,norintermsofthecombi-nationofwhichfactorsmayprovetocombinemosteffectivelywithotherstocreatedemand,

elementsofeachareconsideredtobenecessaryforeffectiveuptake.

Onthesupplyside,criticalfactorscanalsobecategorizedintofour,includingaframeworkoflegaloradministrativepolicyforevaluationorpublicpolicyresearch;thepresenceofsystemsfordesigning,commissioningand/orconductinganddisseminatingevaluationsorresearch;thecapacitytoevaluationthroughastrongsocialscienceaca-demiaandconsultancysector;andinthenascentphasesofeconomicdevelopment,thepresenceofexternalassistancetofinancesuchanalyticalwork.

Supplyanddemandelementsarenotmutuallyexclusive.Whiledemandisconsideredcriti-caltouptake,italsoreliesupon,andcanbestrengthenedbyanadequateframeworkandsystemforsupply5.

Poverty analysis and its impact on public policyTheGovernmentofUgandastartedtoproducepovertymonitoringdatain1992,throughtheUganda National Household Survey(UNHS)reports,andhassinceupdatedthisinformationeverytwotothreeyears. Thisdata,however,didnotplayavitalrolein assessingpublicpolicyuntilthelaunchofthePoverty Eradication Action Plan(PEAP)in1997.Heavyinvestmentsinedu-cationandhealthservicedeliverythroughthe

4 Weiss,1999,“Theinterfacebetweenevaluationandpublicpol-icy,”Evaluation5:468;Bamburger,2009,Institutionalizing Impact Evaluation within the Framework of the Monitoring and Evaluation System,WorldBank;GaarderandBriceno,2010,InstitutionalizationofGovernmentEvaluation:BalancingTrade-Offs,Working Paper 8, International Initiative for Impact Evaluation;WeyrauchandLangou,2011,“Soundexpectations:fromimpactevaluationstopolicychange”,Working Paper 12, International Initiative for Impact Evaluation.5 Thefullnormativeframeworkcanbefoundatthepaperpre-sentedatthe2012AmericanEvaluationAssociationofthesamename.

PEAPmadeitnecessaryfortheGovernmenttoassesscloselytheusefulnessoftheseinterven-tionsintermsoftransformingsocietywelfare.Hence,thepolicyenvironmentandactorswereopentotheproductionofreportsthatwould,ineffect,illustratehowfarthenationalpolicywasbeingeffective.

In1999,theGovernmentofUgandadesignedapovertymonitoringstrategythatguidedtheproductionofbiannualpovertystatusreports,andassociatedmonitoringreportsandpublica-tions.ThePoverty Status Reports(PSRs)werehighqualityanalyticalpieceswhichdrewuponquantitativeandqualitativehouseholdandfacili-tiessurveydatatodeterminethepatternsandchangesinruralandurbanpoverty.


Theinstitutionsresponsibleforgenerating,ana-lyzingandreportingpovertydatawerecriticaltothisprocess,notonlytheireffectivenessasstand-aloneinstitutions,butalsofortheinter-relationshipsbetweentheseagencies.ThePSRswerereliantongood,regularstatisticaldataproduction,andtheworkoftheUgandaBureauofStatistics(UBOS)wascriticalinthis.UBOS,withsubstantialfinancingfromtheWorldBank,initiatedaprocessofproducingregularnation-widehouseholdsurveysonhouseholdincomeandpoverty,healthstatus,populationtrends,andthenlater,onothereconomicandpublicpolicyissues.

WithintheMinistryofFinance,PlanningandEconomicDevelopment(MFPED),aPovertyMonitoringandAnalysisUnit(PMAU)wasestablished, tomonitor, analyse and reporton data generated on poverty and relatedissues,includingthepreparationofPSRs(withfinancingfromUK).Thisworkwassupportedby the Government-sponsored Economic

PolicyResearchCentre(EPRC).TheUgandaParticipatory Poverty Assessment Process(UPPAP)wasalsoestablishedinMFPEDtopro-videqualitativedataonkeysocialeconomicindi-catorsandtheimpactsofselectedGovernmentpoliciesonthecitizenry.ThequalitativedatawasanintegralpartoftheinformationusedtopreparePSRs.

Eachoftheseestablishments:UBOS,PMAU,UPPAPandEPRChadhighlyqualified,com-mittedandmotivatedindividualsinkeyposi-tions.Theability toproducehighqualitymonitoringreports ina timelymanner forpolitical andadministrative consumptionref lectednotonlytheir individualabilities,but their willingness towork together todeliverdemanddrivenmonitoringreports.WithinMFPED,astrongworkingrelation-shipbetweenPMAUandthetopmanagementof theMinistrymeant that findings fromtheseproductsmadetheirwayintopolicyandallocativedecisions,and in turn,attractedincreasedofficialdevelopmentassistance.ManyoftheissuesraisedinthePSRsinf lu-enceddecision-makingatbothattheCabinetand theParliamentary level,andhelped infocusingexpendituresinareasthatweremostmeaningfulforpovertyreduction.

ThewindowofopportunityandpracticeintheproductionofPSRsinUgandareflectedthepri-orityandprogressmadeinpovertyreductionfromthePresidentdown,andtherelationshipsandabilitiesoftheinstitutionsandindividualsinvolved.ThistookplacethroughoutthefirstandthesecondPEAP(1997-2000;2000-2003)but,declinedwiththethirdPEAP(2004-07).AnalysisoftheextenttowhichthesecondPEAPreflectedthefindingsofevaluativeresearchwas

The window of opportunity and practice in the produc-tion of PSRs in Uganda reflected the priority and progress made in poverty reduction from the President down, and the relation-ships and abilities of the institutions and individuals involved.


producedinthesecondUPPAPcarriedoutin2002,findingthatthePEAPdidindeedplacegreateremphasisoncross-cuttingissues,suchasenvironment,andrecognizedtheheterogeneityofthepoorinthegreateremphasisplacedonthedecentralizeddeliveryofservices,andondistrictlevelplans.Inamoresubtlemanner,whilethe1998/99UPPAPraisedconcernoverthenega-tiveimpactonthepoorofcostsharinginhealthservices,thiswasnotaddressedinthePEAPof2000,butwasannouncedbytheGovernmentin2001duringtheelectioncampaign(OPM,20086).Morebroadly,thetimingofthePEAPcycles,revisedin2000and2004,didnotmatchwiththeelectioncyclesof2001and2006,which

hasresultedinsomeanalyticallydrivenanalysisappearinginthePEAPoremanatingduringorfollowinganelection,suchastheStrategicExportInitiative,whichwasinitiatedin2001followingtheelection,butwasnotevidentinthePEAP.

ThedeclineinthePEAP’sinfluenceduringitsthirdphase(2004-07)occurredasthepublicpolicydebateondevelopmentwithinthecountryandamongstinternationalstakeholdersshiftedtowardseconomicgrowthandenhancingtheaccountabilityofthestate,inthefaceofevidenceofmalpracticeandcorruptionintheuseofthestate’sresources.

Shift to budget and performance monitoring for accountabilityWhilst theevidencefromhouseholdsurveysandPSRsbegantorevealthattheoverallpov-ertyheadcountwasreducinginthecountry,itwasalsobeingnotedthatgrowthanddevelop-mentwasbecomingincreasinglyimbalanced(MFPED,2005;UBOS,20067).Asthepublicpurseexpanded,basedbothonastrongandstableeconomicgrowthrate,relativelylowinfla-tion,andaconsiderableriseinofficialdevelop-mentassistance,sotoodidconcernsontheappli-cationandaccountabilityforpublicspending.Effortstostrengthenpublicfinancialmanage-mentincludedtherecognizedneedtounder-standhowpublicresourceswerebeingappliedcentrally,andlocallyunderthedecentralizedsystemofgovernmentandpublicprogrammeimplementation,introducedin1997.

6 OfficeofPrimeMinister,2008,IndependentEvaluationofUganda’sPovertyEradicationActionPlan1997-2007,Vol2:PoliticalEconomy,OxfordPolicyManagementLtd.GovernmentofUganda.7 MinistryofFinance,PlanningandEconomicDevelopment,2005,PovertyStatusReport2005,GovernmentofUganda;UgandaBureauofStatistics,2006,UgandaNationalHouseholdSurvey2005/06-Socio-EconomicModuleReport,GovernmentofUganda.

Theemphasisonmonitoringshiftedawayfromperiodicanalysisofpovertytrendsandcauses,andtowardsthemonitoringofbudgetspend-ing.Duringthelatterhalfofthelastdecade,theMFPEDintroducedaseriesofreformstoenableMinistries,DepartmentsandAgencies(MDAs)andLocalGovernments(LGs)toplanandbudgetannuallyaccordingtoclearbudgetlines,andagainsttheprovisionofproductsandservices.Systemshavebeenintroducedrequir-ingallMDAsandLGstoreportquarterlyonspendingandprogresstowardsstatedoutput(productandservice)targets,asthebasisforfuturefinancialreleases.

ThismassivereorganizationandgrowthintheadministrationofGovernmentplacedincreasingattentiononthegenerationanduseofadminis-trativedataandstatistics,andonthemonitoringandoversightmechanismsinplacetocaptureandreportonperformanceinformation.


Thepoliticalinterestandpressuretomonitorspendingandresultshasincreasedsincethere-introductionofmulti-partypoliticsinUgandain2006,andthegrowingattentionofthedomesticmediaandinternationalcommunitytounequalgrowth,andtheincidenceofcorruptionintheuseofpublicresources.ThePresidentandotherseniorpolicymakershavetakencognizanceoftheseissues,andhaveplacedincreasingdemandsonthepublicservicetoimproveitsstewardshipofresourcesandensureeffectivedevelopment.

Theimpactofthisonpublicinstitutionsisstillunfolding.Effortstoimproveoversightinkeyservicedeliveryinstitutions(throughregularimplementationandbudgetmonitoring),andthroughinspectionofservicedelivery,haveincreased,thoughnotinauniformorconsistentmanner.TheformerPMAUintheMFPEDhasbeentransformedintoaBudgetMonitoringandAccountabilityUnit(BMAU)totrackexpendi-tureandoutputsagainstbudgetsandplannedactivitiesinasampleoffrontlineinstitutions,throughdirectfieldmonitoringandreport-ing.MonitoringisfocusedontheoutputsthatareagreedandsignedupinthePerformanceContractsbetweenthePermanentSecretaryofMFPEDandtheimplementingagencies.Effortstoreorganizetheoverallinspectionfunction

ofGovernmentareunderway.APresidentialdirectivetoinitiatepublicforatoholdlocalGovernment to account (so called public‘barazas’)hasbeeninitiatedbytheOfficeofthePrimeMinister,andtheUBOSisseekingtoexpanditsmandatetoimprovethequalityofadministrativestatistics.

Simultaneously,theOfficeofthePrimeMinister(OPM)hasreinvigorateditsownconstitutionalroleoncoordinatingtheimplementationofpub-licpoliciesandprogrammesthroughestablishingarobustmonitoringcoordinationandoversightfunction.Buildingonanearlyattemptatpro-ducinganoutcome-basedreviewofthePEAPin2007,bi-annualGovernmentPerformanceReportswere initiated in theFinancialYear2008/09andtwo-dayretreatsofallMinisters,PermanentSecretariesestablishedunderthePresidenttodiscusstheperformancereport,holderportfolioMinisterstoaccount,andpro-posecorrectivemeasures.Since2011,thesebi-annualretreatshavebeenexpandedtoincludeallLocalGovernmentCouncilChairpersonsandChiefAdministrativeOfficers.Thishasexpandedthebasisofdebatearoundpublicservicedelivery.

Inthisnewenvironment,accountabilityhasbecomethecentralconcern,withtheconsequentde-emphasisongeneratinginformationforthepurposesofunderstandingcausesandgenerat-ingpolicylessons.TheconsiderabledifferencesinpracticesacrossGovernmentinthemonitor-ingandinspectionofpublicinvestmentsreflectsthebalanceofpriorities,incentivesandcapaci-tiesacrossthesectors,asinfluencedin-partbytheinternationalcommunitywhocontinuetoinvestincertainsectorsoverothers(notablythefrontlineservicessuchashealth,education,waterandincreasinglyroads).

The emphasis on monitoring shifted away from periodic analysis of poverty trends and causes, and towards the monitoring of budget spending.


Theeffectivenessoftheincreasedmonitoringofpublicspendinghasyettobebornout.TheregularpublicpresentationofinformationontheperformanceofGovernmenthasnotyetappearedtohaveimpactedonpublicpolicy,buthasresultedinthegreaterpublicfocusontheneedtoenforceaccountabilities,andsignifi-cantlyhasalsorevealedthewidespreadmisuseof

funds..However,giventhebreadthanddepthofevidenceonchallengestopublicservicedelivery,thepoliticalclassandlegislativearmhavestilltomakebestuseofthisinformationinshiftingpolicydirections,reallocatingresourcestomoreefficientareas,orincasesofmisuseofresources,holdingculpritstoaccount.

Efforts to strengthen the analytical and the evaluativeTheconcertedeffortstostrengthenmonitoringhavecomeatacost.Thepracticeofpublicsec-torevaluationhasneverbeeninstitutionalizedinthecountry,butwasreasonablywellservicedinthelate1990sandearly2000sthroughthePSRsandotheranalyticaltoolsandproducts.Thishasledtoadeficitintheanalysisofresultsandconstraints,andinthe identificationofpolicylessonsandchoices.Monitoringitselfdoesnotanswerthesequestionsoraddresstheseissues.

Between2005and2008,areviewofevaluationinUgandafoundeighty-fiveevaluationscom-missioned,ofwhichtenwerecommissionedorco-managedbytheGovernment(OPM,20098).Whenreviewingthese tenreports indetail,severalwerefoundnottomeetbasicqualitystandardsforevaluation,andweresubsequentlyreclassified as performance assessments orreviews.Intermsofcoverageofpublicinvest-ments,Governmentestimatedin2009thatlessthan10percentofprojectsovertheperiod2005-089werebeingsubjectedtoevaluation.Inasam-pleofMinistries,DepartmentsandAgencies,thesamereviewfoundlittleexplicitdemand

8 OfficeofthePrimeMinister,2009,“Mappingevaluationprac-tice,demandandrelatedcapacity”,IanC.Davies.UnpublishedReport.GovernmentofUganda.9 Includingdonor-financedprojectsimplementedthroughthepublicsector.

forevaluations,alignedwithweakorganiza-tionalcapacityandlimiteduseofthosethatwereconducted.Inanapparentcontradiction,itwasalsofoundthatwasaperceivedneedfor‘moreevaluation’inprincipal(ibid),reflectingnotweakdemandperse,butthelackofaclearpolicy,incentivesandresourcestoevaluate.

Oftheevaluationsthatwereconductedduringthatperiod,thereislittleevidenceoftheirimpactduetoalackofappraisalbyGovernmentortheinternationalpartnersondisseminationoruse.Anexceptionweretheevaluationscoveringtheagriculturesector,startingwithonecoveringthePlanfortheModernizationofAgriculture(PMA)in2005,asecondlookingattheperfor-manceoftheNationalAgriculturalAdvisoryandDevelopmentService(NAADS)in2008,andathirdbeinganimpactevaluationalsoonNAADSin2009.Eachoftheseindependentevaluationsgaveabroadlypositiveassessmentofprogress,withtheimpactevaluationon2009showingpositiveresultsonadoptionofimprovedtechnologies, productivity and per capitaincomes.Thestudy(Benin,200910)alsofoundthatbetween2004-08,NAADSwasassociated

10 BeninS.(2009).Impacts of and Returns to Public Investment in Agricultural Extension: the Case of the NAADS Programme in Uganda.IFPRIResearchReport.Washington:InternationalFoodPolicyResearchInstitute.


withanaverageof24-53percentincreaseinpercapitaagriculturalincomeoftheprogramme’sdirect participants compared to their non-participantcounterparts.However,asnotedbyothercommentators,thatdespitereportedsuccessesofNAADS,overallindicatorsforagri-culturalgrowthwerenotimproving(KjaerandJoughin,201111).ThishaspresentedaproblemforGovernment,andresultedinchangingstrate-giesonagriculture,andNAADS,includingtherenationalizationofextensionworkers,despitethefindingfromthe2008evaluationthat ‘areturntousingpublicsectorextensionwork-ersforthemajorityofserviceswasnotaviableoption’(Ibid,2011).Thisraisestwoissues,onerelatestotheunitofanalysisoftheevaluation,whereNAADSasaninitiativemaybeseentoberelativelysuccessful,butdoesnottakeintothebroadercontext,whichmaybelesspositive,andhencepointsatmorefundamentalstructuralchallenges.Thesecond,reflectstheuseofevalua-tiveevidenceincaseswhetherthemajorityofthepopulation,includingpolicymakers,havedirectpersonalknowledgeaslandownersandfarmers,andwherethemajorityoftheelectorateliveof

11 Kjaer,A.M.&Joughin,J.,2012,“TheReversalofAgriculturalReforminUganda:OwnershipandValues”,Policy and Society,Vol31,Issue4,November2012:pp.319-330.

theland,andthusrequireevidenceofeffortstoimprovetheirlot.

Thepaucityofrigorouspublicsectorevaluationshasbeenidentifiedasaconstrainttoimprovingthecultureofdebatingempiricalevidenceinpublicpolicy.Toaddressthis,theOfficeofthePrimeMinisterbegananinitiativetostrengthentheframeworkandproductionofrigorousevalu-ationsacrossthepublicservice.Startingin2008,OPMledonthedesign,implementationanddisseminationofevaluationsofthesuccessesandfailuresofthePEAPoverthedecadeofimplementation,andtwo(2008and2011)onimplementationoftheParisDeclarationonAidEffectivenessinUganda.

ThePEAPevaluationprocesswasimportantinthatitwasmanagedbyasteeringcommitteecomposedofrepresentativesfromOPM,MFPEDandNPA,aswellasfromthefundingdonoragencies,andwasagoodexampleofhowinter-ministrycoordinationcanwork,iftherewasaspecificfocusortask.ItwasalsoimportantinthatOPMunderstoodhowtheevaluationresultscouldandshouldbedisseminatedandactedupon,managingaseriesofworkshopsforvariousstakeholders,writingawhitepaperforCabinet,basedontheevaluationresultsandgovernmentresponse.Whilesomewhatsup-plydriveninorigin,theevaluationdidfindanaudienceamongstpolicymakers,withthefind-ingsandrecommendationsdiscussedtwicebyCabinet,andhaveinturninfluencedtheshapeofthePEAP’ssuccessor,thefive-yearNationalDevelopmentPlan.

Subsequenteffortstostrengthenevaluationprac-ticeincludethedevelopmentofanationalpolicyonmonitoringandevaluationwhichdefines

The paucity of rigorous public sector evaluations has been identified as a constraint to improving the culture of debating empirical evidence in public policy.


theroles,requirementsandpracticestoembedintothepublicservice,approvedbyCabinetinMay2013.ThePolicyoutlinesthedelineatedfunctionsofmonitoringandevaluation,andprovidesaprescriptionfortheestablishmentofaGovernmentEvaluationFacility (GEF).PreparationfortheGEFbeganinparalleltothePolicyin2010,withtheFacilitylaunchedin2011.TheSecretariatoftheGEFislocatedattheOPM,andincludesthecomponentsofarollingnationalevaluationagendadeterminedbyCabinet;avirtualfundtoprovidereliablefinancingforevaluationsselected;andanationalsub-committeeonevaluationcomposedofrep-resentativesofGovernment,academia,NGOsectorandthedonorcommunitytooverseethedesign,production,qualityassuranceanddis-seminationofthestudies.

AsofJuly2013,theGEFhascompletedtwomajorpublicpolicyevaluationsontheeffective-nessofGovernment’sresponsetoabsenteeisminthepublicserviceandtheeffectivenessofpublicprocurementanddisposalsauthorityandhasapipelineofsixfurthermajorpublicpolicyevaluations,coveringavarietyofpub-licservicedeliveryrelatedtopics, including;theeffectivenessofGovernment’semploymentstrategy,acomparativeevaluationofpublicandprivateservicedelivery;theimpactofthelandactamendmentsonillegallandevictions,theimpactofaspectsoftheNorthernUgandaSocialActionFund,.EachevaluationismanagedbyacentralcoordinatingMinistry,eitherOPM,MFPEDorNPA,withevaluationsconductedinhouse,oroutsourcedtoresearchorcon-sultancyinstitutionsdependingonthetopicandcapacity.Allevaluationsaresubjectedtoindependentreferencegroupsforqualityassur-ance,andCabinetpapersarewrittencontaining

thefindingstofacilitateuptake.Governmentresponsesarerequiredtoallevaluation,build-ingontheexperienceofdraftingaCabinetWhitePapertotheindependentevaluationofthePEAPin2008,wheretheactionsfromtheevaluationwererigorouslyfollowed-up.

Thestrengtheningofsupplyislinkedbacktodemandbyseniorcivilservantsandpoliticianstorevitalizedsomeofthecoordinationstruc-tureswithinGovernment.AchangeinMinisterandPermanentSecretaryinOPMin2009ledtotherenewalofthenationalcoordinationframe-workofcommitteesestablishedbyCabinetin2003,butleftdormantintheinterveningperiod.AthreetierstructureofcommitteeslinkCabinetwithcross-sectoraltechnocrats,andprovideaconduitforfeedingGovernment-widedirectivesdowntoimplementers,andfeedingevidencefromanalysisback.Anationalmonitoringandevaluationtechnicalworkinggroupmeetsbi-monthlyandwiderepresentationfromacrossGovernment,NGOanddonorcommunities.Sub-committeesonevaluation,andonmoni-toringandoversighttakeupmuchoftheworkwhichfeedsbackintotheworkinggroupandontoCabinet.

ThisstrengtheningofCabinetleadcoordina-tionsystemisongoing.AfeatureofUganda’spublic sectorgovernancearrangementshashistoricallybeenthepowerofthePresidencyandtherelativeweaknessoftheCabinetsys-tem.ApracticalconsequenceisthatMinistersarenotsubjectedtoacollectivegovernmentdiscipline.ThereisnoclearmeansthereforeofholdingMDAsresponsibleandaccountablefortheirperformance.Themainleverforcol-lectivedisciplineisthewithholdingoffundingbyMFPED,butitoftencannotbeappliedto


coregovernmentservices,andis ineffectiveagainstMDAswithpowerfulpoliticalbacking.ThemovesbyMinistersandsubsequentlythe

appointmentofanewandpoliticallypowerfulPrimeMinister followingthenationalelec-tionsof2011havebeenimportantmovesbytheGovernmenttofillouttheroleprovidedforthePrimeMinisterintheConstitution.TheCommitteestructureisamajorstepforwardinstrengtheningtheCabinet’sroleinmain-tainingastrategicandcollectivedemandforperformancetowhichportfolioMinistersandtheirMDAsaresubordinate.Itthusprovidesthespaceforempiricalevidencetobeconsideredanddiscussedatahighlevel,andthusgiveagreaterchanceforuptakeinpublicpolicyandimplementation.

The withdrawal of donor financing and the increasing role of politics in civil administrationThesereformstookabackwardstepinlate2012whenacaseofgrandcorruptionwasidentifiedintheOfficeofthePrimeMinisterandotherpartsofGovernment,resultinginalarge-scalewithdrawalofdonorfinancialaidandbudgetsupport.Thishadtwoimmediateimpactsontheevaluationagenda.First,themajorityshareof therecurrentanddevelopmentbudgetofthemonitoringandevaluationactivitieswasfinancedbydonors,andthustheaidfreezehadanimmediateandbigimpactonstaffingwithnumbersreducingbyoverhalfinthedepart-ment12andthemajorityofactivitiesstopped,includingseveralevaluations.Second,thecred-ibilityoftheOfficecameintoquestionwithinvestigationsintoseniorandmid-levelstaff.Duringthisperiodofinvestigation,courtcasesandinternalreorganization,theinstabilityand

12 Contractstaffrepresentedapproximately80%ofallstaffintheDepartmentofM&E,OPM.Ofthese,70%hadcontractswithOPM,allofwhichwerefinancedthroughdonorfunding.WiththefreezeofofficialaidtoOPM,theGovernmentcancelledthecontractsofthesestaffaftertwomonthsinDecember2012.

lackoffinancesresultedinthestallingoftheevaluationagenda.

WiderquestionsregardingthelegitimacyoftheOPMtoeffectivelyplaysitsconstitutionallymandatedroleofleadinggovernmentbusinessandcoordinatinggovernmentpoliciesandpro-grammeshavebeenraised—thoughtoagreaterextentbydonorsthaninternallywithinthepub-licservice.Withthemovementofseniorcivilservants,includingthePermanentSecretary,andprogressmadeGovernment-wideinaddressinganumberoffinancialcontrolsandaccountabilityissues,supportisnowreturning.Thisincludestotheevaluationandevidenceagenda.However,theformofthisassistanceislikelytochange.Financialaidislikelytobeprovidedinexcep-tionalcircumstanceswherecontrolsarestrongandalternativemodesofdeliveryinappropriate,sodonorsarelikelytoreturntoproject-typesupportthroughself-managedprocurementoftechnicalassistanceandservicesforevaluation.Thisislikelytoachievethenecessaryreduction

A feature of Uganda’s public sector governance arrange-ments has historically been the power of the Presidency and the relative weakness of the Cabinet system.


infiduciaryrisk,butalsoreducepublicsectorownership,andpossiblycommitment.

Asabreakpoint,thecorruptioncaseandthefreezingofaidenabledobserverstoseetheextenttowhichtheGovernmentiscommittedtothemonitoringandevaluationagenda,havingtofinancealloperationsfromitsowncorebudget.Government monitoring, the performancereportandCabinetretreatshavecontinued,albeitwithsomechallengestoqualityposedbythesubstantialreductionintrainedstaff.Theevaluationagendahassuffered,withdwindlingnumbersattendingthecross-Governmentevalu-ationcommitteeanddespiteahealthypipelineofevaluations,slowprogressbeingregistered.

TheintroductionofanewPermanentSecretarytotheOfficeofthePrimeMinisterinJune2013willchangethelandscapeagainandopportu-nitiesforevaluationinthepublicsector.Earlysignsofpromisearethatshehasthepoliticalsupporttoshake-upthepersonnelandsystems,withanumberofchangesintheleadership,

auditandprocurementstaff.Itwilltaketimetoestablishherselfandtackleaseriesofissuesbeforeturningtoeffortstostrengthentheevalu-ationfunction.

Thepoliticalspaceforevidence-informeddebateinUgandaappearstobereducing.Theopencontestationofideaswithintherulingparty,particularlyamongst theyoungermembers,havebeencontrolled,andnewlegislation,suchasthepublicordermanagementbill,isreflec-tiveofattemptsbytheGovernmenttocontroloppositionandpublicrallies.Reformswithinthepublicserviceremainpiecemealinsequencingandfinancing,andconsequentlytherearemajorchallengesineducation,healthandinfrastruc-tureprovision.Inthiscontext,decision-makingisincreasingbasedonallegianceordefensivebases,andnottoharnessinnovationornewapproaches.Thispositionsevaluationfirmlyonsupply-sidewithinthepublicsector,withlim-itedopportunitiesforgrowth.However,greaterdemandmayexistamongstnon-stateactorstoamplythevoicesofthepublic.

ConclusionTheexperienceofUgandaoverthepast twodecadeshasillustratedthattheestablishmentandeffectivenessofmonitoringandevaluationpracticesaremostacutewhenpolicies,insti-tutionsandindividualactorscometogether.Thecompositionandbalanceofthesefactorsshiftovertime.Thepoverty-focusedanalyticalworkwaspre-eminentinthelate1990s/early2000satatimewhentherewaspoliticalcon-sensusovertherequireddirectionofchangewithinthenewGovernmentandwithstrongsupportfromthedonorcommunity.Thus,theanalyticalworkhadawillingaudiencewithin

thepoliticalclassandamongstthecountry’ssupporters.

AsthePEAPbegantolosetraction,andtheshiftinGovernmentpolicymovedtowardseco-nomicgrowth,sotootherewasareductioninthesupplyanddemandofpovertyanalysis,andatacticalshiftwithMFPEDtowardsmonitor-ingandexpendituretracking.Thenumberofagenciesinvolvedinmonitoringandoversighthasproliferated,asdemandpressuresandsup-plyopportunitieswithinthepublicsectorhaveincreased.


Inthe lastthreeyears,notwithstandingthecontinuedfocusonmonitoringforaccount-ability, theOPMhas led the drive towardsreintroducingmore analytical work in thepolicydebatethroughtheestablishmentoftheGovernmentEvaluationFacility.ShiftsinthepoliticaleconomyoftheCabinet,withanewandpowerfulPrimeMinisterandareinvigorated

coordinationmechanismappearedtoprovidingabasisandstructurethroughwhichdemandforevidencecouldbeelicitedtoinformpublicservicedelivery.

However,thedemand-sideconditionsconsid-eredprerequisitesforeffectiveevaluationuptakeanduseseemtobedecreasingmorerecentlyinUganda.Theopennessofthepoliticalspacefordebateisreducing,asthemovetowardsthenextelectioncampaigninadvanceof2016nears.Theorganizationalsystemsandtheindividualchampionsareinsomecasesstillpresent,butareunablethriveorfunctionaseffectivelyastheyshouldwhenunderminedbycorruptioncases,potentiallossoflegitimacyandinthecontextofcontinuinglypoorlyperformingpublicservices.Theopportunitiesforevaluationtoinfluencedecisionsmaynowlieprimarilyoutsidethepublicsectorandwiththepublicthemselves,supportingcitizenstodemandbetterservicesandrights.

David Rider Smith is the UK Department for International Development’s Evaluation Adviser in Uganda and Acting Head of the Capacity and Quality Group in its Evaluation Department. Prior to joining DFID in 2012, he worked for five years in the Uganda Bureau of Statistics and the Office of the Prime Minister in Uganda as an advisor to the Government, leading on the consolidation of performance monitoring, Cabinet reporting, strengthening district statistics and setting up the Government Evaluation Facility. He has previous worked as an evaluation specialist in the Evaluation Office of UNDP, as a researcher at the Natural Resources Institute, and as a survey manager for USAID-supported programmes in Mozambique. His background is in geography, sociology and development economics.

The opportunities for evalu-ation to influence decisions may now lie primarily outside the public sector and with the public themselves, supporting citizens to demand better services and rights.


Dr. El Hadji Gueye, BA, MA, PhD, Coordinator of CLEAR Initiative for Francophone Africa

strengthening national evaluation systemsisimportantforgoodgovernance,improvementoftheeffectivenessofpublicpolicy,andbet-termanagementofdevelopmentwork.Transparencyintheallocationandmanagementofstatefundsandofficialdevelopmentassistance,theobligationtobeaccountable,andtheemergenceofacultureofresultsaresomeofthemanyreasonsthatjustifytheimportanceofcreatingandencouraginganevaluativecultureinthecountriesofFrancophoneAfrica.

Thedemandisthere.Ifitisnotsupportedbynationalgovernments,itisstronglyencouragedbytechnicalandfinancialpartners.Itisincreasinglyastrongcondition-alityinofficialdevelopmentassistance.However,theevaluationofpublicpolicies,programs,andprojectstodrawlessonsthatcanhelpguidedevelopmentactionsisnotaneasyexercise.

Learningfromtheimplementationofdevelopmentactivitiesisnotsystematic.Ineffectiveapproachesandmethodologiesandlaxproceduresarerepeatedthroughignoranceorlackofcommunicationbetweendevelopmentactors.Countriessufferfromthelackofnationalexpertscapableofundertakingconclusiveevaluationstudies.Itisthereforecrucialtodevelopacomprehensiveprogramofcapacitybuildingorempowermentofdevelopmentactorsforbetterappropriationofthecultureofevaluationandharmo-nizationofapproaches,concepts,andmethodology.Thiscanimprovetheleadershipofthestates,andinevitablyrequirestheestablishmentofanationwideresults-basedassessmentsystem.

ItisimportanttoinvesttimeandeffortinNECDbecausetechnicalcapacitytoprovidegood-qualityassessmentisinadequate.However,inacontextmarkedbyweakpoliticalcommitmenttoanationalsystemofmonitoringandevaluation(M&E),poor-qualityassessmentsdiscouragegovernmentinitiative.Itisthereforeurgentthatwecreateacriticalmassofevaluativecompetenciestorootevaluationineverydaypracticeand

Strengthening National Evaluation Systems—

from CLEAr Francophone Africa


giveFrancophoneAfricancountriesthecapac-itytobettermanagetheirdevelopmentefforts.

Tohavenationalexperts ineachcountry isallthemoreimportantbecausethemissionsconductedbyexpatriateevaluatorsoftenleadtoirrelevantconclusionsandrecommendations,sincetheseexpertsarenotfamiliarwiththerealitiesofthelocalculture.Andtheresultsoftheseevaluationsareoftennotincludedinnationaldecisions.Anotherreasonistheneedtocapitalizeontheexperiencegainedfrompreviousmissionsandbuildupaninforma-tionbaseofgoodpracticesandpastmistakestoavoid.This informationcanthenbeusedinthedesignofeffectivenationalandsectoralpolicies.

Lessons learned with respect to NECDSuccesses, failuresUnfortunately,littlehasbeenimplementedsuc-cessfully.Thisisoftenduetoalackofpoliticalwill.Toachieveownershipofevaluationasarealtoolthathelpsindecisionmaking,suchbehav-iorshouldbeencouragedandsupportedbythecentralgovernment,butalsobytheparliament,whichhasafundamentalroletoplay.

Theestablishmentofanenablinglegalframe-workisaprerequisite.Butonitsown,itisnotenough.Itshouldbefollowedbyraisingtheawarenessofdevelopmentactorsatalllevelsbytrainingevaluationprofessionalsandcreatingamotivatinginstitutionalframeworkformorerigorousevaluations.

Anothersourceoffailureliesintheconceptoftheindependenceoftheevaluator.Theunbridled

powerstruggles(withstrongpoliticalinvolve-ment)thattakeplaceforcesomeevaluatorstomakeunethicalcompromisestopleasepoliti-cians,donors,andcivilsocietyleaders.Nationalassessmentbodiesmusthavetheautonomynec-essaryfortheexerciseoftheirduties.

ChallengesThemainchallengeissuccessfuladvocacyforagoodevaluationcultureinallFrancophoneAfricancountries.Weneedtobringpoliticiansandholdersofpublicpowerintotheevaluationdynamic;makeevaluationanintegralpartofthepolicy-making,planning,andbudgetcycle;andencouragetheuseofassessmentresultsandgoodgovernance.

Anotherchallengeishowtosuccessfullydealwithconflictsofinterestandtheinfluenceof

Unfortunately, little has been implemented successfully. This is often due to a lack of political will. To achieve ownership of evaluation as a real tool that helps in decision making, such behavior should be encouraged and supported by the central government, but also by the parliament, which has a fundamental role to play.


politicalpowerandnotallowtheresultsoftheassessmenttobedivertedbythestronginvolve-mentofpolitics.

Good practices, tips• Implementafavorableandmotivatinginsti-

tutionalframeworkforevaluationatalllevelsofpublicadministration.

• Developnationalassessmentpoliciesandmonitortheirrigorousenforcement.

• Raisetheawarenessofmanagersinpublicadministrationaboutthecultureofevaluation.

• Supportinstitutionsspecializedintrain-ing,consultancy,andcapacitybuildinginevaluation.

• Oneshouldfocusonaction,practice,andfieldworkratherthanusethetraditionalapproach.

• Takebabystepsandconsolidategainsbeforemovingtothenextlevel.

• Involvecivilsocietyandensurethatthereisamechanismforcitizenparticipation.

• Create internationalnetworksofevalua-tionprofessionalstopromotethesharingofexperiencesandexchangeswithaviewtoharmonizingpractices.

Examples of models of national evaluation systems that are performing well Ghana’sexamplehasalottoteachus,asdoestheSouthAfricanmodel.Benin’smodelcanalsobeusedasanexample,althoughmuchremainstobedonethere.

Modelsforconsideration:

• ThePublicExpenditureManagementandFinancialAccountabilityReview(PEMFAR),basedonthepublicexpenditureandfinancialaccountability(PEFA)indicators

• SNISE(NationalIntegratedMonitoringandEvaluationSystem),whichistheframeworkforallgovernmentplanning.

Institutions involved inNECD include thefollowing:

• Benin:InstituteforEmpiricalResearchinPoliticalEconomy.

• Benin: TheOffice of Public Evaluation(BEPP)inthePrimeMinistersOffice,theDepartment for Projects and ProgramsMonitoringintheMinistryforDevelopment.

• BurkinaFaso:TheDepartmentofTrainingandEducationalInnovationin2IE,whichprovidescontinuousprofessionaldevelop-mentprogramsinM&Eforprofessionalsandorganizations.

• Burkina Faso: International Society ofSustainabilityProfessionals.

• Côte d’Ivoire: Global Partnership forEducation.

• Gabon:ISTA.

• Senegal: African Centre for AdvancedStudies inManagement(CESAG),whichprovidestrainingandseminarsinM&Eofpublicpoliciesandprograms.


• Senegal:TheCLEARInitiativeforFranco-phoneAfrica;CESAG,whichprovidestrain-ingforcapacitybuildingindevelopmentofresults-orientedM&E,supportandguidancetostatesonthedevelopmentandimplemen-tationofpoliciesinM&Eatnationalandsectorallevels,innovationsandcasestudies

onM&E,andthedisseminationofresultsandknowledgeinM&E.

• Senegal: The General Delegation forStateReform(DREAT), theDepartmentofNationalPlanning in theMinistryofEconomyandFinance.

The road ahead Urgent issues in national evaluation capac-ity development include the following:Theindependenceofnationalevaluatorsandthepositioningofthegovernmentstructuresinchargeofevaluation.

Weshould:

• Workforthedefinitionofanationalassess-mentpolicyforallFrancophoneAfricancountries.

• Establishacoordinationunitforallevalua-tionactivities.

• Createaregionalcrucibleformeetingsandexchangesofallprofessionalstosharebestpractices.

• CreateplanningandM&Eunitsinthecentraladministrationandinministrydepartments.

• Extendadvocacyforanevaluationculturetocivilsocietyandlocalelectedofficials.

• Makeprovisionsinthestatebudgetforthefinancingofcapacitybuildinginevaluation.

• Developaneffectiveprocessforselectingthebestprofessionalprofiles,activeintheresults-productionchainandintheconduct

ofpolicies,projects,andprogramsandequipthemwith the approaches andmethodsneeded to conduct successful evaluationmissions.

Key elements for successful national evalu-ation capacity developmentFirst, commitment by the state to efficientnationalM&Eandeffectiveinvolvementofthestate,alongsidedonors.Thisshouldbefollowedbythedefinitionofalegalframeworkforthesystematicevaluationofpublicpolicies,projects,andprograms.

Otherusefulsteps:

• Demystifyevaluationandinstitutionalizeit.

• Createnationalstructuresforevaluationatthecountrylevelorreviveexistingstruc-tures;associate themwiththeactionsofCLEAR.

• Establishstructurestopromoteevaluationandorganizeannualsymposia,highlysci-entificlectures,anddiscussionsonmethodsandconceptsforthebestuseofthetools.

• Contributetothedevelopmentofnationalpoliciesbasedonlessonslearned.


Dr. El Hadji Gueye’s expertise includes methods of impact evaluation for programs and projects, M&E for health expenditures, management based results, and economic evaluation for programs. He has provided consulting services for the World Bank, UNICEF, UNAIDS, UNDP and technical assistance on data collection and analysis for evaluation and management based on the results with Marginal Budgeting for Bottlenecks tool in Senegal, Burkina Faso, and Gabon.

He has coordinated and facilitated workshops on National Health Account, M&E for health programs, and strategic planning for HIV programs. He is the head of the Economics Department and the coordinator of CLEAR Initiative for Francophone Africa at the Centre Africa in d’Etudes Superieures en Gestion (CESAG) in Dakar, Senegal.

He has a PhD in econometrics and statistics, a master sin econometrics, and a bachelors in quan-titative methods at the faculty of Economics and Social Sciences, University of Geneva, and a bachelor in Mathematics from UCAD in Senegal.

• Introduceevaluationinacademictrainingprograms.

• Createadynamicregionalnetworkofevalu-ationpractitioners.

Actions that would strengthen evaluation capacity developmentThefundamentalmeasuresare:

i) Advocateforacultureofevaluationamonggovernments,policymakers,anddevelop-mentactorsatalllevels.

ii) Carryouttrainingandcapacitybuildingtocreateapanelineachcountryconsistingoflocalexperts,abletoadequatelyconductassessmentmissions

iii) Providesupportandadvicetothebodiesinvolvedintheconductofdevelopmentpolicyandprograms,aswellasthoseinpol-icy-makingpositionsinthestatemachineryandinparliament,civilsociety,anddecen-tralizedagencies.

iv) Createadegreecourseinevaluationandprovidefundingforappliedresearchinthefieldofevaluationinordertochangepractices; develop new tools that aremoreadaptedtotheparticularcontextofFrancophoneAfrica.

v) Establishanationalcoordinationstructureforevaluationactivitiesineachcountry.

vi) Develop networking and exchangesbetweenpractitioners.

vii) Organizeconferences,symposia,andsemi-narsoncurrentissuesinevaluationwiththeobjectivesofinforming,raisingaware-ness,andharmonizingpractices.

viii)Disseminateresults.



developing an evaluation culturewithintheBankandinregionalmembercountriesisoneofthecorner-stones of the independent evaluationpolicyrecentlysubmittedforapprovalto theCommitteeonOperationsandDevelopmentEffectiveness(CODE).Thisobjectiveisbasedonthenotionthatpro-motinganevaluationculturewithinaninstitutionsuchastheAfricanDevelopmentBankplacesanemphasisonevaluationoutcomesandonthelessonslearnedfromsuccessesaswellasfailures.Aninstitutionwithanevaluationcultureassumesthatitsmembers:

• Agreetouseevaluationoutcomesandlessonslearned.

• Haveasufficientunderstandingofwhytheinstitutionusesevaluationoutcomes.

• Candevelopandimplementauto-evaluationsystems.

• Useevaluationoutcomeswiththeintentionofsupportinginstitutionalchangeanddevelopment.

Promoting an evaluation cultureTheBankanditsstakeholdersseektopromoteacultureofevaluatingoutcomes,whichcanonlyoccurifevaluationisatthecenterofthiseffort.DespitetheincreasingengagementinfavorofoutcomesalreadyrecordedwithintheBank,promotinganevaluationculturepresupposesalargeprogram.TheOperationsEvaluation(OPEV)Departmentwillhavetocollaboratewithotherpartnersinthiseffortandensurethatitsworkcompletesotherinitiatives,suchasthequalityassuranceworkdonebytheBank’sQualityAssurance&ResultsDepartment(ORQR),anaturalstrategicpartnerinachievingthisobjective.

Toward an Evaluation Culture in the AfDB: Practices and

Challenges

Mohamed Manai, Manager, OPEV, AfDB


Reinforcing stakeholder engagementIt is important tomention that engagementoccursinthreemainareas:

• WithintheframeworkoftheBank’sprojectcycles.Byexaminingtheefficiencyandeffec-tivenessoftheQualityAssuranceReviewsforprojects,policies,andstrategiesduringtheinitialphasesofconceptionanddevelop-ment,andbyquestioningthemeritofthequalitystandardsused,OPEVoffersinstitu-tionallessonsandpracticesandbringsinpastexperiencestoreinforcethesepreliminaryqualityassurancemechanisms.Furthermore,inevaluatingthepoliciesandsector-basedstrategiesofgroupedevaluations,OPEVexaminestheextenttowhichpastlessonshavebeentakenintoaccountinsaidpoliciesorsector-basedstrategiesineachcountry.Similarly,invalidatingProjectCompletionReports,OPEVpresentthesector-basedorregionalgoverningbodieswiththemeritsofthefindingsfromthevalidationandtheProjectCompletionReports.Thisensuresoptimallearningforstafffromtherelevantregionalandsectoraldepartments.

• Within the framework of individual evalu-ations.Engagingmanagementandrelevantpersonnelatcriticalmomentsoftheevalua-tioncycle(evaluationframeworkdocument,identificationofstakeholderandreferencegroups,disseminationofknowledgefromfindingsinthefieldandfrominteractionswithbeneficiaries) isabestpractice that

hasbeenadoptedbyOPEVaspartofthedevelopmentandimplementationofevalu-ationsofprojectsandofsector-orcountry-basedstrategies.Thisengagementgenerallyinvolvesdialoguearoundevaluationobjec-tives,evaluationquestions,approaches,andmethodologiesusedtojudgefactorsofsuc-cessandfailure,aswellasthecommunica-tionsandknowledge-sharingstrategies.

• WithintheframeworkofplanningOPEV’swork program.Consultationswithmanage-mentandwiththeCommitteeonOperationsandDevelopmentEffectiveness (CODE),whichtakeplaceduringthedevelopmentofindependentevaluationtherollingworkpro-gram,seeksstakeholderbuy-inforevaluationthemestoincreasetheirusefulness.Notonlydoesthisapproachfavortheemergenceoftangibledemandsfromdecisionmakersanddevelopment-programplannerssearchingforlessonsandexperiences,butitalsofostersthedevelopmentofanevaluativeculturebasedontheuseofoutcomesandfeedback.

OPEV’sengagementmustgobeyondthecon-finesoftheBank.First,itmustfosterinterac-tionsbetweenbeneficiarycountriesbyinclud-ingthemasmuchaspossibleinevaluationsofprojects(thematic,sector-orcountry-based);second,itneedstodevelopcooperationwiththeinternationalevaluationcommunityandotherorganizationsinterestedinsupportingevaluationinAfrica.

Increasing the impact of evaluations in regional member countries Thereisincreasingdesireforevaluationknowl-edgeinAfDBregionalmembercountries.MakingsurethattheknowledgegeneratedbyOPEVis

madeavailabletothemisanindispensablestepinensuringthatevaluationsareimpactfulanduseful.Withinthisframework,OPEVpromotes

thedevelopmentofanevaluationcultureinthreeways:

• Regional member country participation in the evaluation process.Participatingincountrystrategyevaluationsisencouragedbutnoteasilyachievableduetocountries’lowcapacities.Nationalstakeholderinvolvementthroughouttheevaluationprocessmustbeaccompaniedbyawareness-raisingsessionsonthemeritsofnationalevaluationsystems,withinaframeworkofimplementingaman-agementapproachthatisbasedondevelop-mentresultsandtherationaluseofpublicresources.

• Raising awareness, by targeting regional member countries in communication efforts.Knowledge-sharingsessions,con-ductedinregionalresourcecenters,facilitateparticipationbyregionalmembercountries.TherecentworkshoporganizedattheAfDBSouthernAfricaResourceCenter(SARC)atteststothehighlevelofinterestofregionalcountryrepresentatives,who,inthecaseoftheSARC,madesubstantivecontributionsonsubjectssuchasprivatesectordevelop-ment,Bankassistancetoregionalintegra-tion,economicandsector-basedreviews,and

thedevelopmentofgeneralinfrastructure(transportationinparticular).

• Supporting regional member country efforts to develop national evaluation capacities.Regionalmembercountriesarealsolookingtogenerateevaluativeknowl-edgefromtheirownprogramsanddevel-opmentprojects,whethertheyarefinancedexternally or not.They are increasinglyinterestedinreinforcingnationalevalua-tioncapacities.OPEVhasalreadybeguntoundertakereinforcementactivitiesincoun-trieswhereatangibleneedhasbeendem-onstrated(suchasEthiopiaandTanzania),focusingitseffortsonministriesinchargeofplanningbyensuringthepracticalandhands-ontrainingofpersonnelinchargeofpost-evaluationmonitoringfornationaldevelopment programs. In otherwords,OPEVwillsupportthedevelopmentofanevaluationcultureinregionalmembercoun-triesbysupportingtheirevaluationsystems.Suchsystemshelpensurethatevaluationsmakeadifferenceinpoliciesandinpractice.

OPEVisintheprocessofdeepeningitspartner-shipswithotherorganizationsintheinterna-tionalevaluationcommunity,anditcontinues

Evaluation Learning Event in SArc


toparticipateinmajorinternationalevaluationcommunitiessuchastheevaluationnetworkof theDevelopmentAidCommittee of theOrganisationforEconomicCo-operationandDevelopment,andtheEvaluationCooperationGroup.OPEVcontinuestodevelopstrongandmutuallybeneficialrelationswithbilateralaid

entitiesandengageswithorganizationsfocusedonevaluation,suchastheAfricanEvaluationAssociation,variousregionalcentersforlearningfromevaluationsandoutcomes,EvalPartners,the International Initiative for EvaluationImpact,andphilanthropicorganizationsinter-estedinsupportingevaluationefforts.

Managing and disseminating knowledgeOPEVusesvariousmediumsandtechnologiestofosterknowledgesharing:

• Evaluation reports,conceivedtoconveykeymessagesclearlyandsimplyarepairedwithotherevaluationpublications(“eVALUationMatters”)andshort,adhocmemosthatflagimportantissuesforthebenefitoftheBoardofDirectorsandseniormanagement.

• The OPEV website,thedepartment’sprimaryinterfacewiththeworldhasbeenredesignedtobemoreuser-friendlyandtoallowuserstoquicklylocatetheknowledgeproductstheyneed.

• A database of lessons learned, whichwillallowusers to search for informationbythemeorbycountry.OPEVwillcollectles-sonsfromotherorganizationstomaximizetheusefulnessofthelessonsdatabase.ThissystemwillincludeindependentevaluationsaswellasCompletionReportReviews,andit

willcoverawiderangeofevaluationprod-uctsandlessons.

• Face-to-face interactions withtheBank’soperationsstaffwillalsobereinforced.TheEvaluationCommunityofPractice1willbeexpandedandwillalso targetoperationspersonnelwhen appropriate.OPEVwillorganizeotherknowledgeeventstoreachalargeraudience(suchasmanagementandnationaloffices,aswellasotherstaffinvolvedinprojectandprogramconception,imple-mentation,monitoring,policy,andqualityassurance).

• Regional member country awareness willbeincreasedthroughregionalworkshopsandvideoconferences,whichallowfordissemina-tionanddiscussionofknowledgegeneratedbyevaluationwork.Tofurthertheseefforts,OPEVwillworkinclosecollaborationwithcountryteamsandwiththeBank’sregionalresourcecentersandoffices.

Implementing evaluation recommendationsDiscussingtheimplicationsofevaluationout-comesforfuturepoliciesandoperationsisessen-tialforpromotinganevaluationcultureandensuringthatevaluationrecommendationsareeffectivelyimplemented.Thisunderscorestheimportanceofreportingandsharingknowledge

andestablishesalinkbetweenevaluativeknowl-edgeandsustainedevaluationofdevelopment.AmechanismtostrengthenmanagementactionplansforimplementingrecommendationsfromOPEVwillbelaunchedshortly.



Suchamechanismwillmonitorthewaymanage-mentreceivesrecommendations,aswellastheclarityofitsresponsethroughpreciseactionsandinitiativesspacedovertime.Theentitiesinvolvedinreportingmanagementactionswillensurethat

actionstakenaretransparentandthatCODEisinformed.Itisequallycrucialtomonitortheimplementationofevaluationrecommendationstoensurethecredibilityandusefulnessofinde-pendentevaluations.

Promoting an evaluation culture within the Bank: constraints and challengesThemain constraints and challenges are asfollows:

• Levelofsupportfromalllevelsofmanage-mentforauto-evaluationandtheseriousnesswithwhich recommendationsfrominde-pendentevaluationresultsareimplemented.Thequalitycontrolprocessforauto-evalua-tionproductsintheBank’sactiveportfolioneedstobereinforcedtoreducethegapthathasbeennotedbetweenauto-evaluationsandindependentevaluationinexaminingportfolioperformance.

• Levelofsupportfortheauditingandqualityassurancebodiesinadecentralizedenviron-ment.Commonstandardsofqualityneedtobe linked to lessons fromexperiencesorlessonsthathaveservedasthebasisforthedesignofnewpolicies,strategies,andoperations.

1 OPEV’sEvaluationCommunityofPracticewaslaunchedin2011asaninternalforumforsharingknowledge.In2012,thecom-munitygrewintoaforumforsharinganddiscussingevaluationknowledgewithoperationspersonnelworkingonrelevantissues.

• Level of support forbodies in chargeofinspectionsandinternalauditsfortheircata-lyticroleinfosteringanevaluationculture.

• Strategicpartnershipsbetweenfunctionalentities inchargeof evaluations,qualitycontrol,andstrategiesorpolicies.Suchapartnershipwilltranslateintoapoolingofresources,withtheintentionofdevelopingcapacity-buildingprogramsandlearningworkshopsbasedonoutcomesfrompolicyandstrategyevaluations.

Mohamed Manai is Division Manager in the Operations Evaluation Department of the AfDB. Before joining the Bank, Mohamed worked as a Management Consultant for PriceWaterhouse in Kuwait, Tunisia and Morocco. He is trained in finance and holds an MBA degree.


Dr. Sukai Prom-Jackson, Inspector, Joint Inspection Unit, United Nations

ConceptualFramework for Evaluation and National Evaluation Capacity Development1

Why Africa Has Not Developed to Its Full PotentialMoussa Seye (African savant) on challenges to African growth and development (August 7, 2013) (translated from Wolof)

—If someone lends you his eyes, you will only see what he wants you to see. Theeyesarethewindowstothemind.Theywilldirectwhatyouthink.

—If someone lends you his language, you will use the models he has developed to express reality based on his own ecological and historical background. Thereisnosovereignthathasdevelopedonthelanguageofanotherculture.

—If someone lends you his mind based on how he defines value and truth, you will work for it and continue to fix it to fit.Youwillneverknowandyouwillneverbecomeafithumanbeing.

—Ifyoudonotfarmwhatyoueat,andyoudonoteatwhatyoufarm,youwillnevergaincontrolofyourappetiteandstomach.

1 Thisnoteisafollow-uptoreflectionsaffordedbytheopportunitytoparticipateinthe2012AfricanThoughtsLeadersForuminBellagio,organizedbytheCenterforLearningonEvaluationandResults.Itincorporatesdialogues,reflections,andnotesbasedonconsultationswithseveralthoughtleadersinAfrica.ItprovidestheconceptualframeworkthatshouldguidethedevelopmentoftheevaluationfunctioninAfrica.Follow-upnotesincludeusingtheframeworktoanalyze:(i)thestateofaffairsinAfrica,(ii)supportforevaluationcapacitydevelopmentbytheUnitedNationssystemsanddevelopmentpartners,and(iii)whatisneededbynationalstomovetheprocessandengageallpertinentnationalinstitutionsinvolvedincriticalthinkingandaction.

Rebirth, Restoration, Reclamation, and Responsibilities of the Evaluation

Function of Africa


IntroductionMoussaSeye’swords embodylessonsabouttheunderlyingproblemsofdevelopmentinAfrica.Theyalsoapplytoevaluationandnationalevalu-ationcapacitydevelopment.

Evaluationandsupportforevaluationcapac-itydevelopmentshouldavoidbeingpartoftheproblemofthedevelopmentofAfricabyusinglenses, language,models,andimportedbest-practicemodelsthatarenothome-grown.Toplayaconstructiverole,thetheoryandpracticeofevaluationinAfricaandsupportfornationalevaluationcapacitydevelopmentmust:

• UnderstandtheaddedvalueofevaluationforAfrica.

• Beconsciouslysensitiveandtakeinspirationfrominheritedlegacies.

• Confrontanddirectthecurrentvarying,oftenhybrid,andcomplexrealitiesofthepresent.

• BuildthefuturesandsecuritiesforAfricainaglobalcontext.

Evaluationhastobegroundedinaclearunder-standingofthegoalsbeingarticulatedfortherebirth,restoration,reclamation,andrespon-sibilitiesofdevelopmentforAfrica.ThesegoalspresentanalternativeoneillustratedMoussaSeye’swords.

Thispaperpresentsthecorecapacitiesrequiredforevaluationandaconceptualframeworktosupportnationalcapacitydevelopment thatisfoundedonprinciplesfortherebirth,res-toration,reclamation,andresponsibilitiesofevaluationforAfrica.Thepapercovers:evalu-ationanditsfundamentalcapacities,capacitiesforevaluationrequiredforaneffectiveroleinthedevelopmentofAfrica,andaconceptualframework fornational evaluationcapacitydevelopment.Follow-upworkisplannedforusingthisframeworktoassesswhatishappen-ingwithevaluationinAfricaandthecapacitydevelopmentsupportprovidedbydevelopmentpartners.

EvaluationandCapacitiesforCriticalThoughtEvaluationisjudgmentofvalueorworthandprovidesinformationtosupportdecisionmak-ing. Indevelopment evaluation, it supportsaccountabilityfortheeffectiveuseofresources,lessonsforimprovement,knowledgesharing,andthedistillationofthisknowledgeforusewithinorganizationsandinthewiderglobalcommunity.

The main factor that makes evaluationstandout fromotheroversight functions in

organizations is thecritical mode of think-ing it requires. It is a rigorous process ofinquiry.Itexaminesactionsandresultsandasks thequestions:Arewedoing the rightthings?Arewedoing thingsright?Arewegettingresults thatmakeadifference?Arethesetherightresults,andwhatistheimpactandvalue?Whatexplainsthedifferencemadeandwhatenhancesimpactandsustainability? In providing answers to these questions,evaluationusesthesystematicmethodsand


toolsofbasicscientific inquiry.Assuch, itdrawsoncriticalreasoningskillsandappliesbothdivergentandconvergentmodelsofrea-soningandanalysis. Italsousesprinciplesfrommanagementsciencetoenhanceutility.It thusabidesbyasetofdefinednormsandstandards—basedonavaluationsystem—thatseektoenhance thedevelopmentofstrongevidencethatiscredibleandtrustworthy(thatis,impartial,valid,andreliable),andtheuseofsuchevidenceandrecommendationsfromtheevaluation.

Toenhancecredibilityandutilityinsupport-ingdecision-makingprocessesassociatedwithaccountability, improvement, learning, andknowledgemanagement, governments andorganizations have put in place evaluationfunctionswithdefinedstructures,institutionalframeworks, and systems andmechanisms.Figure1providesanillustrationoftheevalua-tionfunctionusingthekeyelementsofdemand, supply, and resultsandtheirrespectivecapabili-tiesasaframework.2

Evaluation Capacity Development and role of Evaluation for the Development of Africa National evaluation capacity development isanendogenousnationalprocess todevelopthecapacity for“evaluative thinking.”Thisencompasses theusecritical thinkinganddivergentandconvergentreasoningtopro-ducecredible(impartial,valid,andreliable)andtrustworthyevidencetosupport(i)deci-sionmakingand(ii)actionthatwouldbenefitnationalgrowthanddevelopment,includingthedevelopmentof social and intellectualcapital.Thisnationalprocess iscarriedoutatalllevelsofsocietyandisinstitutionalized,inparticularaspartofthebusinessmodelforeducationandschooling(formaland infor-mal),governmentandnationalgovernance,universities,thinktanksandresearchcenters,theacademiccommunity,andtheportionsof

theprivatesectorinvolvedininnovationanddevelopment.

ThiscapacityforevaluativethinkingismademoreurgentasthepeopleofAfricaconfrontthecurrentandemergentglobalchallenges,forwhichexistingblueprintsarenotalwaysadequate,andthedevelopmentchallengesofAfricatiedtoAfrica’sneedtotakea leader-shiproleindecisionmakingandactionwithconsciousreflection,resilience,andadaptabilitytoaddresssixkeyframingconditions,outlinedinBox1.

2 TheframeworkillustratedwasdevelopedbytheJointInspectionUnitoftheUNsystem.See“AnalysisoftheEvaluationFunctionoftheUNSystem”(2013).ItbuildsonpreviousworkbytheauthorasstaffoftheWorldBank.


Figure 1: The Evaluation Function—Key Components and Systems Framework for National Evaluation Capacity Development

-

Demand and Intentionality for Evaluation- REBIRTH National context and drivers of demand Nature and level of demand Vision and strategy for evaluation

Supply, Adaptation, Growth of Function-RESTORATION

Results of Function Utility and Impact RECLAMATION

CREDIBILITY: IMPARTIALITY AND BALANCED PERSPECTIVES

Structural Independence

Functional Independence

Controls against management influence

Professional/Technical Independence

Behavioral Independence

ENABLING

ENVIRONMENT: NATIONAL, ORGANIZATIONAL & INSTITUTIONAL EVALUATION FRAMEWORK

Governance; legislative, executive, management Evaluation architecture and structures at all levels Institutional framework: Policy, Norms,

RELEVANCE, RESPONSIVENESS, EFFICIENCY AND ADAPTABILITY

Coverage Responsiveness challenges Efficiency

Adaptability of function for continuous improvement and growth

CONDITIONS IN PLACE TO ENHANCE USE

Report Quality

Dissemination and communication strategy

Timeliness

Accessibility

Transparency

Follow-up mechanisms

EFFECT OF USE

Effect of use on changes or development

Value for money

Mutual accountability

Learning nations and communities

Contribution to knowledge for development and evaluation

NATURE AND LEVEL OF USE

Recommendation implementation rates

Use of strategic decision-setting

Corporate use

Formative Use and Improvement

Box1:SixmajorframingconditionsforevaluationinAfrica• Africa’slong-termgrowthprospectsarestrong,propelledbybothexternaltrendsinthe

globaleconomyandinternalchangesinthecontinent’ssocietiesandeconomies.• ResourcewealthandtheresponsibilityformanagingthiswealthintheinterestofAfrica.• Theshiftintheglobaleconomiccenterofgravity,theroleofemergentglobalpartners,and

theresponsibilitytoremainstronganddrawfromlessonstoengagenewdominantpartners,suchasChina.

• Thewaveofdemocratizationandcitizenactivismwithinacontextofadominantandgrowingyouthpopulation,highratesofemployment,andextensiveopencommunicationsystems.

• Theeventualitiesofthefutureasaffectedbyglobalization,climatechange,economicandpoliticalinstability,anddemandsforrapidresponseatalllevels(individual,organizational,andgovernmental).

• Thesalienceoftechnologiesandinnovationandtheroleofhome-growntechnologiesandmethodsintheformulationofappropriatemodelsthatare“Africa-centered,drivenbyAfricaninterests,androotedinAfricanframesofreference.”

Sources:ZendaandRobMoore;Roxburghandothers2010;Jeng2012.


ToenhanceitsroleinAfricaasacriticalinstru-menttoaddresstheseemergingchangesandimperatives, evaluationmust seek tonotbepartoftheproblemofdevelopmentinAfrica,butrathertoprovidesolutionstoproblemsandcontributetothefutureofgrowthanddevelop-mentandtheultimategoalsofdignity,peace,andprosperity.

Approachestoaddressingtheseframingcondi-tionsmayvary.Itissubmittedthatforanysug-gestedframework—includingthoseforevalua-tion—toserveasacriticaltoolortooutlinethecapacitiesneededtoguidenationaldevelopment,itmusthavethefollowingfourelementsforahome-grownframeworkofaction.3Thesearereferredtointherestofthepaperasthe four Rs(4Rs)

Rebirth:Africa’sexperiencessinceindepend-encesuggestthatdevelopmentandevaluation

3 Theseconstituents,describedbyAbouJengandSukaiProm-Jackson,effectivelycapturetheviewsofseveralpartiesconsultedonwhatwouldmakeevaluationinAfricaausefultoolfordevelopment.

techniques,theories,andpracticeshavebeenacentralpartofAfrica’sproblems.Theycannotthereforecontinuetobethesoleframeworksuponwhichsolutionsareoffered.Anewhome-grownapproachspecifictoAfricaisneeded.

Thefirstpriorityofthisapproachmustberebirth. Themeaningofrebirthmayvary,butwhatitentailsisabreakfromthepastanditsdisempow-eringfeatures.Thisisnecessarytoallowspaceforideas,practices,androutinesthatarenotladenwithnegativelegacies.However, for there to be a meaningful break, rebirth must be total, purposeful, and home-grown. It is given expression in the French phrase: enracinement puis overture.

Restoration:Africa’sencountershaveexposedandsubjectedittoagonies,violation,anddep-ersonalization.There is an urgent need forrestoration:restorationofpride;restorationofheritageindevelopmentnarratives;restorationofambition;andrestorationofgenuineleader-ship.Butitisimportanttonotethatrestoration isnotaboutnorms-localizationorintroducingforeignmodels.Rather, it is about putting back the missing links that were compromised by the dominance of Eurocentrism in Africa.

Reclamation:Thefaultlinesincurrentdevel-opmentmodelsandevaluationmethodsmeanthatAfricamustnowturntoandexplorehome-grownknowledgesystems.ThisisnecessaryforthereclamationofAfricanspacesandownershipindevelopmentnarratives.ApotentialbenefitisthepossibilityofseeingAfricathroughatotallynewlensthataddspride,confidence,andrel-evance tohome-grownknowledge systems.Achieving this requires conceiving methods and models that are particular to, and reflective of, African sensibilities.

The first priority of this approach must be rebirth. The meaning of rebirth may vary, but what it entails is a break from the past and its disempowering features. This is necessary to allow space for ideas, practices, and routines that are not laden with negative legacies.


Responsibility: In rebirth, restoration, andreclamation,thereispotentialintheformula-tionofanAfrica-centered,Africa-specific,andAfrica-drivenevaluationfunctionandmod-elsforevaluation.ThesethreeRsalsoprovidetheframeworkfornationalevaluationcapacitydevelopment.Howtomakethishappenisacriticalconsideration.ItrequiresleadershipandactionfromAfricans.Whatisdifferentthistimeinnationalownershipisthenatureofthevisionthatwoulddrivetheprocess—onethatisfocusedontheinterestsofAfrica.Thisvisiondrawson

thelegaciesandconsidersrebirth,restoration,andreclamationwithdueregardtothevastnessofAfrica,theinterlinkageswithvariouscultures,thecurrentcontexts,andthesustainabilityoftheprogressthathasbeenmade.

Drawingfromthisbackgroundonthe4Rs,thefollowingpassageprovidesguidingprinciplesthatmustbetakenintoconsiderationintheconceptualizationanddevelopmentoftheevalu-ationfunctionandinstrengtheningnationalcapacitiesforevaluationinAfrica.

ConceptualFrameworkfortheEvaluationFunctionandCapacities for Evaluation in AfricaDemand, supply, results for accountability, improvement, and knowledge management within a well-defined, integrated framework that is “Africa-centered”Definition:Evaluationisa judgmentof thevalueorworthofanentity.Theroleofevalu-ationfordevelopmentofAfricamustincludedefiningwhatisvaluedformovingAfricafor-ward.Evaluationisasystematicprocessbasedonscientificandmanagementprinciples.Theprocessofgeneratingevidenceandknowl-edgemustrespectwaysofknowinginAfrica,includingtheimportanceofactiveinvolvementintheconstruction,management,anduseofknowledge.

Systems model—the national evaluation sys-tem:Giventheroleofevaluationforaccount-ability,improvement,learning,andknowledgemanagementintheinterestofthenation,thenational evaluation function and capacitiesrequiredmustnecessarilyusean integratedandholisticframeworkthatappliesasystemsmodel.Thisincludes:

1. Acknowledgingallkeynationalplayersandinstitutionsinvolvedinthedefinitionof,demandfor,supplyof,anduseofevaluation.Thisincludesnationalsandcivilsocieties,government,educationandacademicinsti-tutions(involvedinproductionofhumanresources,knowledge,andintellectualcapi-tal),researchandanalyticcenters,andthinktanksinvolvedinknowledgeproduction(seeFigure2).

2. Italsoincludesunderstandingtheinterre-lationshipsamongdemand,supplyandusebetweenthenationalsystemandtheregionalandinternationalsystems.Figure2providesanillustrationoftheseinterrelationships.


Figure 2: A systems perspective for national evaluation capacity development

*Supportingevaluationcapacitydevelopment

Guiding principles for the evaluation func-tion: Rebirth, Restoration, Reclamation, and Responsibility: Thefollowingsetofguidingprinciplesfromthe4Rsprovidetheframeworkthatmustbepartoftheconstituentelementsindefiningthekeyelementsofthefunctionofnationalsystemsandthatshouldguidecapacitydevelopmentandsupport.The4Rsareinter-related.However,thewaytheyfitwiththekeyelementsoftheevaluationfunctionfordemand,supply,andresultsisillustratedinFigure1.

Rebirth: Defining demand, purpose, and added value of evaluation.ThevisionforevaluationinAfricamustpromoteevaluationthatinformsdevelopmentthatisAfrica-centeredandrootedinAfricanrealitiesandtheissuesthatarecriticalformovingAfricaintothefuture.

• Insodoing,evaluationmustmovebeyondcurrentpredominantmodelsofknowledgegenerationbuiltonhypothesistestingofimported “bestpractice.” Ithas tovalue

various forms of knowing recognized inAfricansocietiesaswellasinvariouspartsoftheworld.Itsbusinessmodelmustbeinclu-siveinsettinganagendaofcriticalissuesimportantforthefutureofAfrica,address-ingkeychallengesandallaspectsofsocietyandallgroups,beyondthosepertinenttodevelopmentpartners.

• Giventhattherearenoblueprints,evaluationmustfocusondevelopingunderstandingsofboththeprocessesthatinformimprovementinameaningfulwayandtheendresults.Inthisregard,formativeevaluationtakescenterstagealongwithsummativeandaccount-abilityevaluation.

• Thepurposeofdoingevaluationforaccount-abilitymustdepartfromafocusonaccount-abilitytoexternaldonorsandmovetotheaccountabilityofthegovernmenttoitsciti-zens,aswellasaccountabilityofcitizenstoeachother.

Country Nationals

International Partners*

Traditional:Multilateral andBilaterals and donors Non-traditional:South-South; Global Partnerships; Foundations and Funds

- Networks"

Government - Legislature- Executive- Management- Education

and Schooling

- Universities

Non-Government: Academic community, Research/Analytic Centers; Think Tanks; Private Sector; Networks

Regional Partners


• Aparamountgoalofevaluationistobuilddynamic learning groups and learningnationsthatareresilientandadaptabletofast-movingchangeandfocusedontheuseofknowledgeforinnovationandchange.

• EffortsdirectedatknowledgemanagementmustfocusongarneringknowledgeinAfricatodirectandmanageexternalsupportandtocontributetoglobalknowledgeandplatforms.

• Evaluationmustacknowledgethatinaworldofrapidchange,knowledgeisaflow.Itmovesandchanges.Thevalueofanyoneknowl-edgeproductdependsonhowitisdeveloped,used,andsubjectedtoinnovation.Ithasvaluewhenused.

Restoration: Supply guided byrestorationofknowledgebasedontheAfricanlensandphi-losophyandusedtodefinenorms,standards,models,methods,andsensitivitytocontext:

• Movefrommarginalizationtothemain-streamprocessofknowledgeconstruction.EvaluationmusttakeAfricanvoicesintothemainstreamofdevelopment.

• ThefunctionneedstopartakeofAfricanindigenousphilosophiesandwaysofknow-ingandlearning.Thishasimplicationsforapproaches andmethods. Evaluation forAfricamustbebuilttoincorporateindig-enouswaysofknowledgegeneration,judg-ment,anduse.

• EvaluationmustbepoisedtodevelopnormsandstandardsbuiltonAfricanvaluesystemsintheirpureorsyncreticform,asdeemedbestforthefutureofAfrica.

• Givendiversityandvaryingcontextsandcul-tures,itmustincorporatetraditionalvaluesaswellasvaluesinthenexusbetweenAfricaandEurope,AfricaandIslam,Africaandcomplexity,andAfricainamodernworldoffast-movinginformationtechnology.

• Itmusthavespacefortheindependencetomakejudgments.Todothis,onemustreturntotradition—refinementmusttakeplacefirstinone’sculture,andthenintheworld.

Reclamation and Responsibility for Results: EvaluationmustassumeleadershipforAfricaandtakeitsplaceindefininghigh-impactevalu-ationfunctionsthatrespondtovaryingdemandsinabalancedway.Itmustreclaimitspositionintheconstructionofknowledgeandinservingasalegitimateagentofchange.Itshouldseektodevelopavisionforevaluation,withkeyelementsthatincludebalancingthefourfoldpurposeofevaluationforaccountabilitytonationsandtodevelopmentpartners,improvementofpractice,buildinglearningnationsandenhancinginno-vationsinsociety,andcontributingtoglobalknowledgeandtothefieldofevaluation.

• Thevisionshouldnot limitevaluationtoactivitiesbygovernment.Itshouldhighlighttheroleofnationalinstitutionsinvolvedinthesupply,demand,anduseofevaluation,includinggovernment,educationandschool-ing,thinktanksandresearchandanalyticcenters,universitiesandtheacademiccom-munity,theprivatesector,andnationalsandcivilsocietiesatlarge.

• Itshouldseektoenhancethedevelopmentofa learningnationengagedinconstantreflection,criticalreasoning,action,and


innovation.This requiresbuilding resil-ienceandadaptabilityinrespondingtorapidchangeonaninformedbasisandonasolidgroundfordevelopmenteffectivenessandsustainability.

• Ithasaresponsibility toenhancecogni-tivediversityintheactiveconstructionofknowledge.Thisrequiresaninterdisciplinaryapproachthatbringsinvariouscompetencesandengagestheglobaldebatewithvalidevidence.

• Itwillacknowledgecomplexitiesofdevel-opmentandthehybridnatureofvariouscontextsanddevelopmentprogressmadeinAfrica.Thiswillcomplementemergingmethodsandmodelsinevaluation,includ-ingcomplexityscience,adaptivemodel,andmodelsofcognitivediversity.

• Theroleof theevaluator isnot thatofareporter.Rathertheevaluatorisacentralchange agentwho creates room for dia-logue,engagesintherigorousexchangeofknowledge,andincreatingpossibilitiesandsolutions.

• Itmustrecognizeandtakealeadroleinapplyingmoreintegratedandsystemsmod-elsforevaluatingthedevelopmentprogramofpartners.Thisincludestakingaleadroletodefinewhatworkisconductedandtocoor-dinatetheevaluationagendasofdevelop-mentpartnersinalignmentwiththenationalevaluationagenda.

• Itshouldhelpenhanceunderstandingofthe complexityofdevelopmentandhowtraditionalevaluationsofdevelopmentby

partnersaredetrimental todevelopmentimpact.Besidesthenatureofthemodels,thebusinessmodelof“siloevaluations”bydiffer-entpartnersisequallydestructiveandkillsdevelopment:“Whenyoucutanelephantinto17piecestomanageitbetter,youdon’tget17smallermoremanageableelephants.Yougetadead,chopped-upelephants.”4Thelargeerrorsinvolvedinseparateevaluationsonthesamephenomenaraisequestionsofthevalidityofevaluationsbydevelopmentpartners.

• Evaluationshouldconsideranopen-systemsmodel,giventheinterdependenceandnatureofthechangeprocess.5

4 AliMostahari. “SystemsThinkingandDealingwith theComplexityofDevelopment”(Hoboken,NJ:StevensSchoolsofSystemsEngineering,2012).5 Keyfeaturesoftheopensystemsforevaluationcapacitydevel-opmentinclude:

(i)Understandingthesubsystemsoperatingatvariouslevelsofgovernancestructure.

(ii)Emphasizingboththeinternalandtheexternaltheenviron-mentsinwhichthefunctionoperates,sinceeachplaysanequallyimportantroleforquality,impact,andsustainability.

(iii)Sometimessubsystemsrelatetoeachotherinalogicalmanner(inputs,activities,outputs,outcomes,andimpact),butatothertimeschangesinonelevelarelikelytohavenonlinear/causalandoftenunpredictableorunintendedeffects.

(iv)Thisisbecausechangeisaprocessanditispeoplewhochange.Thechangeprocessgoesthroughvariousstagesofmaturityordevelopmentinadoptingguidingprinciplesandsetnormsandstandardsduetoanumberoffactorsthatareembeddedinthefunctionitself—organizationalclimate,natureofdemandanditsevolution,characteristicsofdecisionmakers,leadershipandchampions,organizationalcultureanditsabsorptivecapacityforrisk, learning,improvementorinnovation,externalglobalconditions—allhavearoleindrivingthechangeprocesstowardahigh-qualityandhigh-impactevaluationfunction.

(v)Performanceorlevelofdevelopmentinthesecomponentsandattributesfollowsaprocessofgrowthandmaturation,fromreori-enting,tomechanicaluse,toaroutinelevel,toalevelofconcernforimpactandthesustainabilityofthefunction.

(vi)Thereareidealnormsandstandardssettingoptimalrulesofthegame.Thereisgenerallynoperfectconfigurationofsubsystemsthatshouldbeusedasanideal.Thereisnosuchthingas“onesizefitsall.”Systemsarebestdefinedas”fitforpurpose”ataparticulartime,givenhistoryandcontext.


• Itshouldtakealeadroleindirectingsub-stantialfocusonformativeevaluationtoenhance understanding of what works,why,andhow.Thisis importantfortest-ingdevelopmenthypothesesandfortimelyengineeringandpreventionofthenotionof

developmentasascientificlaboratory.Thecomplexoperationsincomplicatedcontextsandinfrastructureforevaluationmustfocusoncontinuousmonitoring,assessment,andredirection.

referencesJeng,A.2012.“Rebirth,RestorationandReclama-

tionofDevelopment.”Mostahari,A.2012.“SystemsThinkingand

DealingwiththeComplexityofDevelop-ment.”Hoboken,NJ:StevensSchoolsofSystemsEngineering.

Roxburgh,C.,andothers.2010.Lions on the Move—The Progress and Potential of African

Economies.NewYork,NY:McKinseyGlobalInstitute. http://www.mckinsey.com/insights/africa/lions_on_the_move

U.N.JointInspectionUnit.2013.“AnalysisoftheEvaluationFunctionoftheUNSystem.”NewYork,NY:UN.

Sukai Prom-Jackson is an Inspector with the Joint Inspection Unit of the UN System. She has over 20 years of professional experience and leadership in the management and conduct of research and evaluation, in policy formulation and strategic planning and program development, and in the facilitation of learning.

She has worked for the UNDP, World Bank, U.S. government, universities, and research centers in the United States. She has a PhD in educational psychology (1982) with focus on policy research, measurement, and evaluation from Howard University (U.S.).


Dr. Claudia Maldonado, General Coordinator, CLEAR Center for Spanish-Speaking Latin America1

Monitoring and evaluation (M&E)havelongbeenpromotedbythedevelopmentcommunityasnecessaryfortheenhancementofresults-orientedpublicmanagement,tofosteraccountability,andtoenableevidence-basedpolicymak-ingtopromotedevelopmenteffectiveness.Inthedevelopedworld,thistrendresultedintheadoptionofnewpublicmanagementconceptsandprinciplesintheprocessofadministrativereformandmodernization.Inthedevelopingworld,however,thisagendahasbeenmoreforcefullypromotedbythedemandforaccountabilityofdonorsandexternalinstitutions.

Apartfromtheobviousdifferencesbetweenthepolicysystemsandsocioeconomiccontextsofcountrieswithvaryinglevelsofdevelopmentanddemocraticconsolida-tion,itisworthnotingthattheexternalpushforM&Ehasleftanimportantimprintinthescopeandtrajectoryoftheadoptionoftheseinstrumentsintheglobalsouth.ThelowlevelofadoptionandcitizenengagementinthepushforM&Eandtheabsenceofstrong,home-growncommunitiesofpracticeandattentivepublicscommittedtothisagendahaveseverelylimitedthescopefortheincorporationofthelogicofevaluation—andresults-orientedmanagementingeneral—indomesticpolicycirclesbeyondthelifecycleofspecificdevelopmentprojects.Thisis,inpart,becauseinthepastdecade,wehavewitnessedanimportantshiftfromthetraditionalemphasisonprogramevaluation,anchoredinspecificprojectsandsectors,tothecentralroleofthecreationofnationalevaluationcapacitiesandthedevelopmentofnationalevaluationsystemsthataremoreclearlyanchoredindomesticdynamics;localadoption;and,mostimportant,thepresenceoftherealeconomicandpoliticalconditionsneededforsustainabilityanduseforofevaluationfindingsindecisionmaking,acrosspolicyareasandbeyondspecificgovernmenttenures.

1 CLEAR(RegionalCentersforLearningonEvaluationandResults)www.clear-la.cide.edu

Speaking Truth to Power: The Role of National Evaluation

Systems in Democratic Governance


Butwhatexactlyarenationalevaluationcapaci-ties?Howaretheyrelatedtotheconstructionofnationalevaluationsystems?And,mostimpor-tant,whyshouldwecareaboutthisincrediblycomplexendeavor?

InthisarticleIofferanoverviewoftheroleofnationalevaluationsystemsindemocraticgovernanceandreflectonthewindowsofoppor-tunityandthecommonchallengesfacedbyLatinAmerica,whererelativelynewdemocraticregimesinveryunequalsettingsstruggletomeet

demandsforaccountabilityandpolicyefficacy.IdrawonanepistemicconceptofpublicpoliciesandpresentthefunctionalandnormativelinkofM&Esystemswithdemocraticlegitimacyandgovernmentefficacy,andthenpresentsomeexamplesofhowthislinkcancreateavirtu-ouscircleofpolicyreform,asexemplifiedbyrecentdevelopmentsinLatinAmerica.Finally,Ireflectuponthecorechallengesthatmustbemetforthesesystemstobeconsolidatedandtrulybecomesustainablefeaturesofpublicmanage-mentanddemocraticgovernance.

What s Democracy Got to Do With it?Thedevelopmentofnationalevaluationsystemsandtheadoptionoftechnicaltoolstoassessprogramandgovernmentperformancecanbemistakenlyassociatedwithatechnocraticpushforplacingpoliticaldecisionsinhandsofanar-rowgroupofexperts,orthebackdoorforovertpolicyinfluencebyexternalactors.Inmyview,nationalevaluationsystemsrepresentpreciselytheopposite:theyplacebettertoolsandknowl-edgeinthehandsofdemocraticdecisionmak-ing,providinganinformationalpublicgoodforaccountabilityandcitizenempowermentand,ultimately,securingthelegitimacyofdemocraticregimes.

Inademocraticsystem,citizensdelegatesov-ereignpowerstoelectedofficials,whoreceiveamandatetopromotesubstantiveoutcomesforthecommunityandaseriesofinstruments—publicfundsandpowers,authority,regulation,administrative structures, and so forth—toadvancethem.Originally,thepreservationofthismandatecouldtheoreticallybesecuredbytheelectoralincentive(Downs1957)andtheruleoflawoftheWeberianbureaucracy(O´Donnell

2010).Governmentsweredirectlyaccountabletothedemocraticwillbecauseearlydemocraticregimeswhereanchoredinliberalstateswithnar-rowtasks,primarilypreservationofterritoryandtheprotectionofprivateproperty(Gorski2003).

Duringthetwentiethcentury,incontrast,gov-ernmentsincreasinglytookdirectresponsibilityoversocialrealmsthatrequiredamorecom-plexproductionfunction:directinvestmentforproductiveactivities,strengtheningofdomes-ticmarketsthroughregulationsandsubsidies,theexpansionofsocialservices,andaseriesofpoliciesandprogramsaimedatrespondingtopressingsocialdemandsinseveralspheres.Contemporarygovernmentsareresponsibleforanumberofpublicgoodsandservicesthatareessentialforthelivelihoodandwelfareofthepop-ulation,eitherdirectlyorthroughtheirinfluenceonthestructureofopportunitiesforeconomicdevelopmentandthecreationofpublicvalue.

Everyday,decisionsonthedesign,distribution,andmodelsofimplementationofpublicpro-gramsandservicesaffectourlives.Theseinclude


thequalityandrelevanceofpublicserviceswereceive;thestructureofaccesstosocialrightssuchashealthandeducation;thedegreeof(in-)securitythatprevailsinourpublicspaces;and,ingeneral,thestructureofincentivesthathouse-holds,markets,andgovernmentsface.Theseserviceshaveanimportantinfluenceonourcollectiveabilitytomeetbasicneeds,promotedevelopment,andcopewithsocialproblemsofvariouskinds.Forthisreason,understandingtheeffectsofpublicpolicies,andtheattributesthatareassociatedwiththeirsuccessorfailure,aswellasthecoststructureandfunctionalityofthemanagementmodelthatcharacterizesthem,isessentialfordecisionmakinganddemocraticgovernance—anditshouldconcernsusallascitizens.

Furthermore,thescopeofpublicresponsibil-ityhasgonewellbeyondprototypicalbread-and-butterconcerns.Itincreasinglyinvolvesmoreelusive,yetequallyinstrumental(forthedevelopmentofhumancapacitiesandfunction-ing;Sen1999),socialobjectives:futuregenera-tions(sustainabledevelopment),gender,equity,theprotectionofintangibles,theprotectionof

privacy,humandignity,andthird-generationhumanrights,amongothers.Theseaspects,closelyrelatedtogovernmentcapacitiesandperformanceincoresocialareas,haveanimpor-tantinfluenceonourcollectiveabilitytomeetbasicneeds,promotedevelopment,andcopewithsocialproblems.

Nevertheless,thereareimportantobstaclesthatmustbeovercomeforthisidealdemocraticequa-tionofaccountabilitytomaterializeineverydaypolicydebates.First,thereisacollectiveactionproblem.Althoughcitizensmaybevictimsofpoorservicesandpoorgovernmentperformance,itisnoteasyforthemtoorganizeandsuccess-fullymobilizetovoicetheirspecificdemandsandneeds.Inveryunequalcountries,moreaffluentcitizenstendto“votewiththeirfeet”(Tiebout1956)andoptoutofpublicservicesandtheirprovision,therebyunderminingthescopeforthearticulationofbottom-upsocialdemands.

Thepolicyliteraturerepeatedlytellsusthatpro-gramswithpoorordisempoweredconstituen-ciesgraduallybecomemorefragile.Theirqualityislowandtheyarepoorlyfinancedbecausekeystakeholderslackthetime,resources,andinfor-mationtopushforqualityandefficacy.

Inaddition,onthesupplyside,decisionmak-ersfacemultipleincentivesandrestrictionsinadoptingthelogicofresults-orientedmanage-ment.Theycanbebudget-maximizersthatcom-peteforbudgetswiththelogicofbureaucraticcontrolandpower(Niskanen1971);theycanoptforshort-termsolutionsthatprovideimmediatepoliticalbenefitsinsteadofsecuringlong-terminvestmentsthatmaximizepublicvalue;theycaninvestpublicresourcesintheconstructionof clientelistic andcorporatistnetworks for

In other words, we need to track whether the theories of change implicit in any intervention and the conse-quent mobilization of public funds to tackle development challenges are progress-ing toward producing the expected result.


politicalgain;andtheycanalso,justlikecitizens,makeprofoundmistakesintheirdecisionmak-ing(KahnemanandTversky2000).

Furthermore, what we should expect anddemandascitizensarepolicyoutcomes,notsimplypublicprograms.Decisionmakersandgovernmentagenciesproducepolicyoutputs,notnecessarilylinkedtoexpectedpolicyoutcomes.Forthisreason,thelinkbetweengovernmentactionandthetransformationofthecorevari-ablesaroundapublicproblemcannotbeheldasaxiomatic.Aslawyerswouldsay,theburdenofproofisapublicresponsibilityofgovernments,becausecitizenscannotsystematicallygivethebenefitofdoubttopoliticians,withoutstringsattached,andbecausetheopportunitycostsofill-conceivedandpoorlyimplementedpublicinterventionsthatutilizescarcepublicresourcesaretoohigh.

Citizensshouldnotacceptthepresumptionofefficacyorgivethebenefitofthedoubtwhenitcomestoassessingthecontributionofapublic

policytoachievingthesocialobjectivesthatgave rise to it.Anddecisionmakers shouldnotbeentitledtomakethesechoicesbymereintuitionorgoodintentions,inanevidence-freezone.Theactualexistenceandverificationoftheinstrumental-causallinkbetweenpublicchoicesandpublicoutcomesisaninescapablepublicresponsibilityfordemocraticgovernance.Butthislink,ananchorofresults-orientedmanage-ment,needstobeconstructed.Itrequiresentireadministrativeandinformationmanagementprocessesthatspeakthelanguageofresultsandfeedintobureaucraticroutinesandpoliticallanguage.

AsLuisF.Aguilar(2010)hassuggested,thegreatchallengeofthedemocraciesofthetwenty-firstcenturyisnotsomuch“whogoverns”but“howshegoverns,”basedonwhatreasonsandargu-ments;throughwhichinstruments;and,aboveall,withwhatresults.Thisisacoremandateofdemocraticgovernanceinthetwenty-firstcen-tury.Withoutcredibleperformanceandefficacy,democraticlegitimacyisalsoatstake.

AnEpistemicView: TheImportanceofSpeakingTruthtoPowerHughHeclousedtosaythatpublicpoliciesareawaytopuzzleonsociety´sbehalf.Soconceived,publicpolicyisatentativeanswer—andcollec-tivecommitment—tosolvepublicproblemsandpursuesociallydesirablegoals.Ifweconsiderthatpublicpoliciesaretheoriesormodelsforsocialtransformation—hypothesesforsolvingcomplexpublicproblems—andtheyare implementedbypublicorganizationsinconstantinteractionwithanenvironmentthatmovesmuchfasterthangovernmentsanywhereintheworld,theenormouscomplexityanduncertaintyinwhich

thesimplestofinterventionsoperatesrequiresustocreatereliableandtimelytoolstotracktheadequacyandvalidityofthosehypotheses.

Inotherwords,weneedtotrackwhetherthetheoriesofchangeimplicitinanyinterventionandtheconsequentmobilizationofpublicfundstotackledevelopmentchallengesareprogressingtowardproducingtheexpectedresult.Inthatsense,theroleofevaluationsystemsininextri-cablylinkedtothatofknowledgegeneration,dissemination,andlearningasaroutinetask


ofmodernpublicadministration.Soconceived,M&Eallowsforthecreationofalearningcurveinwhichconsciouslyexploratorytrialanderrorinpolicymakingcanpromotebetterprograms,betterchoices,andbetterresultsforthecitizenry.

Whileitistruethatmakingmistakesisashumanasitisinevitable,itisalsotruethatmakingmis-takesinkeydecisionsforthepublicinterest(withpublicfunds)entailsseriousmoralandpoliticalimplications.Inadditiontothedistributionalconsequencesandopportunitycostsoferraticpublicprogramsthatlackaclearroadmapandanexplicitcommitmentforthetransformationofspecificsocialindicators,theerosionoftheeffectivenessandresponsivenessofthestateisamajordangertothelegitimacyandsustain-abilityofdemocraticgovernments.ThisiswhytheM&Eagendaasanewlanguageofpublicadministrationwillallowgovernmentstobetterservecitizens:itwillreduceinformationasym-metriesbetweenrulersandcitizens,ensurethatfiscalresourcesarechanneledinreasonedandreasonablewaysforinvestment,andassurethatpublicprogramsmeetclearlyidentifiedpriori-tiesandaredevelopedwithahighprobabilityofpositivelyaffectingthequalityoflifeofthepopulation.

Doprogramswork?Howdoweknowwhetherprogramswork?Areprogramdiagnosticsaccu-rate?Arepublicinterventionsreachingthepopu-lationtheyareexpectedtoserve?Isthequalityofservicesadequate?Aretherealternativewaystoadvancecertaingoals,atalowerburdentopublicbudgets?Howdoesonedecidebetweencomplextrade-offsintheallocationofscarceresources?Howcanwecollectivelylearnfromexperienceandthatknowledgetoenhancetheeffectsofgovernmentinterventions?Isthereawaytomaximizethebangforthebuck?

Nationalevaluationsystemscananswerthesesortsofquestions.M&Esystemscanaddresstheissues,fromthesmallestdetailofprojectmanagementtobroadstrategicdiscussionsaboutinterventionstrategiesandthegreatdebatesaboutthedevelopmentmodelwewantandhowwecanachieveit.Inthecountriesoftheregion,wehavemademajorprogressinthebasicinsti-tutionalframeworkforassessing,monitoring,andmanagingforresults.However,asAaronWildavskyoncenoted,policyanalysisandevalu-ationareawayofspeakingtruthtopower.ItisthusunderstandablethatM&Esystemscanfacesignificantinternalresistance.Theycanevenbeperceivedasanorganizationalthreattothepublicsectorbydemandingtransparencyandgreateraccountability.Totheextentthatundergoingpublicscrutinyandevaluationcanundermineassurancesgiventhroughbureau-craticroutines,operatinglogics,anddecisionmakingbehindcloseddoors,itisnaturalthatthesesystemshavegeneratedresistance.

Forthisreason,thecreationofanationalevalu-ationsystemmightbenecessary,yetinsufficientonitsowntocreate,promote,andenhancethenationalevaluationcapacitiesthatarerequired

The link between institu-tional frameworks (national evaluation systems) and actual capacity for evalua-tion has been at the heart of institution-building theory and debates.


tostaythecourseandpromotetheutilizationofevaluationresultsbydifferentaudiencesandforseveralinterrelatedpurposes.Thesepurposedincludebudgetcontrol,programimprovement,accountability,policyreform,transparency,andthelike.ForM&Esystemstofulfilltheirmis-sion,thepublicpolicydebatemustbeenrichedwithbetterarguments,evidence,anddiversesourcesofinformationthatarecredibleandwidelydiscussedinthepublicarena.

M&E systems are information systems andprocessesthatareanchoredinresultsandevi-dence-basedinputs.Theyallowbothcitizensandgovernmentstoknowthetruthabouttheeffectivenessandefficiencyofpolicies,tofindoutwhetherpublicactions,instruments,andstrategiesarecorrectorwhetherbasicpremisesandassumptionsshouldberevised.ThatistheprimaryroleofM&Esystems.

National Evaluation Systems and the Quest for Evaluation CapacitiesThe link between institutional frameworks(nationalevaluationsystems)andactualcapacityforevaluationhasbeenattheheartofinstitu-tion-buildingtheoryanddebates.Shouldexist-ingbaselinecapacitiesprecedetheenactmentofformalsystemsforevaluation?Aretheyadirectresultofinstitutionalframeworks,ordotheyalwaysinteractinfarfromlinearrelationships?

Insteadoftryingtosolvethatpuzzle,Iwillpro-posesomelessonsfromrecentLatinAmericantrajectoriesinthedevelopmentofnationaleval-uationsystems.InLatinAmerica, thethirdwaveofdemocratizationcreatedconditionsforarisingdemandforaccountability,trans-parency,andscopeforcitizenparticipationinpublicdecisions.CountriessuchasChile,Colombia,andMexicostandoutforhavingdeveloped strong institutional frameworksaimedat institutionalizingM&Etopromoteresults-orientedbudgetingandbettergovern-mentperformance.Althoughothercountriesintheregionhavefollowedsuitwithdifferingpaceandscope,LatinAmericacanbedescribedasaregioninwhichthebackgroundconditionsandthepoliticalcontexthavebeenconduciveto

theadvancementoftheM&Eagenda.Withinthenextdecadeofso,itisreasonabletoexpectthatallcountries,withthe likelyexceptionsofVenezuelaandCuba,willhavesomesortofnationalevaluationsystemsinplace,asincipi-entasitmightbe.

However,thisimportantstepwillnotbesuf-ficienttosecurestrongandreliableevaluationcapacitiesacrosspolicysectorsandbureaucraticlevels, and extendingbeyondgovernmentalactorsandsmallgroupsof stakeholders.Asmentionedabove,M&Esystemsareafirststepfortheproductionforpublicinformationthatisreliable,relevant,andresults-oriented.Buteffectivecapacitiesforevaluationrequiresystem-levelchangesandpositivefeedbackandlearn-ingcycles,andtheydependonfactorsthatareexternaltotheevaluationsystems.

First,evaluationshavetobeindependentandcredible,andtheymustbereliableasimpartialsourcesofinformation.Forthistobethecase,therehastobeacriticalmassorcommunityofpracticewillingandabletoprovideexternalevaluationservices,and,onthedemandside,


agentscommissioningandoverseeingevaluationmustbewell-versedinpublicadministrationandbeabletoidentifykeyevaluationquestions,coremethodologicalrequirements,anddrafttermsofreferenceandplannedbudgetsaccordingly.

Fortheseconditionstodevelop,theremustbeaclearexternaldemandforresults-orientedinformationthatischanneledintothepoliticalandpolicyprocesswithsomedegreeofsaliency.Ifbothcommissionersandevaluatorsexpecttheproductoftheirevaluationeffortstobeforgotten,neglected,orignored,theywillhavenoreasontoinvestinthequalityandcredibil-ityofevaluation.Furthermore,ifthedecisionmakersresponsibleforkeyperformanceandprocessindicatorsoftheprogramstobeevalu-ateddonotexpectevaluationresultstoresonatewithinthebureaucraticand/orpoliticalarenas,thereisnoreasonforthemtargettheireffortsatinducingbetteroutcomesthatcanbeidentifiedandrecognizedthroughM&E.Inturn,citizensandattentivepublicswhoarenotinformedandawareofthepotentialroleofevaluationresultsandmonitoringdatawillnotunderstandtheirlinktosubstantivedevelopmentpromotion.Withoutsuchunderstanding,theywillfailtoutilizetheseresultsintheirpushforaccount-ability andbetter government responses topublicproblems.

Withthisdescription,Iamnottryingtosuggestthatevaluationcapacitycanonlybedevelopedwhenpublicservants,government,andcitizensarethoroughlyinformed,highlyprofessional,andspiritedandstrongadvocatesofevaluation.WhatIamsuggestingisthatactualcapacitydevelopmentcanbeunderstoodasenendog-enousresultoftheinternalandexternalinter-actionsenabledbynationalevaluationsystems,

butthatstronglydependonotherbackgroundconditions:thepush-and-pullfactorsfortheuseofevaluationinthestandardpoliticsofpoliticalpluralism(democraticgovernance)

AttheCLEARCenterforLatinAmerica,wehaveamandatetopromotecapacitybuildingforevaluation,monitoring,andperformance-basedmanagementtopromotedevelopmentattheregionallevel,aswellastopromoteglobalcross-dissemination of good practices andlearningspacesrelatedtoourcorefunctions(technicalassistance,knowledgegeneration,andtraining).Inordertofulfillourmission,wehaveadoptedasystemicviewoftheroleofcapacity-buildingliketheoneproposedabove:weseektopromotetheunderstanding,adop-tion,andmobilizationofM&Einformationasanewlanguageof thedemocraticpoliti-calinteractionthatisalreadytakingplaceinpluralistsystems,yetdoesnotusuallyoccurinthelanguageofresultsandevidence-basedpolicymaking.Tolowerbarriersofentrytothis language,wepromotetraining,aware-ness,anddevelopmentoftechnicalexpertiseamongwideraudiencesandthediversestake-holderswhoalreadyoccupycenterstage inpolicydebates—civilsociety,activists,watch-dogorganizations,andgovernmentagenciesatthenationalandsubnationallevels—andseektofosterleadershipamongkeystakeholdersinthepromotionofthisagenda.

Byprovidingaccessibleandreliableinformationfromtheregion,weseektopromoteknowledgedisseminationandacceleratelearningprocessesamongpeernations.Furthermore,bymobiliz-ingexistinginformationfromnationalevalua-tionsystemsandspreadingitbeyondtraditionalaudiences,weaimtopromoteuseandadoption


inordertoinformpolicydebatesandhighlighttheimportanceofM&Eforgoodgovernance.

InLatinAmerica,thediffusionofconditionalcashtransfersforpovertyalleviationclearlyhasbeeninfluencedbytheroleofcredibleimpactevaluationandhasalreadyresultedinsuccess-fulpovertyalleviationinterventionsandtheprotectionofotherwiseneglectedconstituencies,suchastheruralpoorandindigenouspopu-lations.Inthehealthsector,informationandcross-learninghaspromotedthedevelopmentofaggressivepublicprogramstopreventcata-strophicexpensesfromout-of-pocketspendingonhealthcare.Inthefieldoffinancialdeepeningandmicrocredit,policydebateshavealsobeenanchoredinrigorousevaluationanddynamiclearningprocesses.Foodsecurityandeducationareotherareasinwhichtheinformationfromevaluationsystemshasbeenpivotalinpromot-ingpolicyreformandaccountability.Althoughnationalevaluationsystemsandcapacityforevaluationarefarfromconsolidated,Ibelievethatthereiswindowofopportunitytopromotethisagendaandfurtheradvanceourcommongoalofdevelopment.

Dr. Claudia Maldonado is currently Professor-Researcher at the Public Administration Division of the Centro de Investigación y Docencia Económicas (Center for Research and Teaching in Economics) a Mexican center for research and higher education specialized in the social sciences. Her research focuses on the political economy of conditional cash transfers in Latin America and comparative development (Mexico and Brazil), social policy and program evaluation.

Prior to her current position, Dr. Maldonado coordinated M&E capacity-building efforts for Mexico’s Consejo Nacional de Evaluación de la Política de Desarrollo Social (National Council for the Evaluation of Social Development Policy, CONEVAL) and worked in the General Direction for Evaluation at Oportunidades, the largest conditional cash transfer program in Mexico

Dr. Maldonado earned her Ph.D in Political Science from the University of Notre Dame, and holds a Master s Degree in Public Affairs from Princeton University.

referencesAguilar,L.2010.“PolíticaPública.México.”

Escuela de Administración Pública delDistritoFederal,SecretaríadeEducacióndelDistritoFederal,SigloXXI.

Downs,A.1957.An Economic Theory of Dem-ocracy.NewYork,NY:HarperCollins.

Gorski,P.2003.The Disciplinary Revolution. Calvinism and the Rise of the State in Early Modern Europe.Chicago,IL:TheUniversityofChicagoPress.

Kahneman,D.,andA.Tversky.2000.Choices, Values and Frames. New York, NY:CambridgeUniversityPress.

Niskanen, W. 1971. Bureaucracy and Representative Government.Chicago,IL:Aldine-Atherton.

O’Donnell,G.2010.Democracy, Agency, and the State: Theory with Comparative Intent. Oxford,U.K.,andNewYork,NY:OxfordUniversityPress.

Sen,A.1999.Development as Freedom.NewYork,NY:Knopf.

Tiebout,C.1956.“APureTheoryofLocalExpen-ditures.”Journal of Political Economy 64 (5):416–24.


Food for Thought• HowtoSpeedupGlobalDevelopment’sLearningCycle

Trevor Davies, KPMG• DevelopingEvaluationCapacity,butWhichCapacity?

Samer Hachem, AfDB• NationalEvaluationCapacityDevelopment:

Past and future Directions, Mohamed Manai, AfDB

• ChecklistforDevelopingaNationalEvaluationSystem

Mohamed Manai, AfDB


Trevor Davies leads KPMG’s International Development Assistance Services Global Center of Excellence

i’ve spent More than20years ininternational development, spendingmostofthattimeinfragileorpost-conflictstates.Inthattime,I’vecometoseeadis-connectbetweenoutcomesandintentions

Weallwanttoensurethe$250billioncommittedeachyeartointernationaldevelop-mentisspentwisely.Yetwedonotlearnenough,orlearnquicklyenough,fromtheworkthat’salreadyhappening—orhappened—ontheground.

Thisknowledge-managementgapmeansweareconstantlymissingopportunitiestoimprovethequalityofourworkanditsimpactontheworld.Severalfactorscontributetothisgap.First,thecycleofevaluationistooslow,anditsfindingsareoftenoutofdateornotrelevantwhentheydoarrive.

Considerthetypicalprojectcycle.Mostdonors—followingtheParisandBusandeclara-tionsonaideffectiveness—wanttoworkmorecloselywiththerecipientgovernmentsandensurethatprojectsarecountry-led.Atthesametime,donorsneedtoensuretheseprogramsmeettheirownstandardsandfitwithintheirownagendas.

Asaresult,theprojectdesignphaseisaniterativeprocess,andittakestime,insomecases,uptooneortwoyears—orevenlonger.Withthedesignphasecomplete,thedonorcanputinplaceabaseline,againstwhichprogresscanbemeasured.Inthemeantime,they

How to Speed Up Global Development’s Learning Cycle

Trevor Davies

… we do not learn enough, or learn quickly enough, from the work that’s already happening—or happened—on the ground… this knowledge management gap means we are constantly missing opportunities to improve the quality of our work and its impact on the world


tenderforasupplierorimplementer,whichcantakeanywherefromthreeto12months,depend-ingonhowthecontractisstructured.Finally,theimplementergetstheprojectunderway.Let’ssayitcontinuesforfiveyears.Whenit’scomplete,apost-projectevaluationiscompleted.

Intotal,that’saneight-yearcycle.Atwhatpointareyoulearninglessonsandfeedingthoseles-sonsbackintotheproject?Ifyoubeginaprojectin2013,youwouldnotlearnlessonsfromituntil2021.Sotheresultsareslowtoarrive,andwhentheydo,theyareconsideredoutofdate.Adonorwillsay:“Well,yes,thatprojectwasflawed,butwedon’tusethatframeworkanymore.”

Andthisisthesecondfactorcontributingtotheknowledge-managementgap:Ourrapidjump-ingfromonedevelopmentconcepttothenext.

InthetwodecadesI’vebeeninvolvedindevel-opment, I’ve seen somany approaches andmethodologies.Atonetime,thenewideawastohavestand-aloneprograms.Atanothertime,itwastohaveaholisticapproach.Thenitwasspotinterventions,thencivilsociety,etc.Theseideaswerestate-of-the-artthinkingatthetime;guidancenotesweresentout,projectsdesignedaroundthem.

Inmanyways,thisrapidevolutioninthink-ingreflectsourgrowingunderstandingofwhatworksandwhatdoesn’t.Butinanothersense,wearesimplycomingupwithnewideastoengageconstituencies—whetherthosearetaxpayersorlawmakersorcheck-writingphilanthropists.Wearefindingnewwaystotellthesamestory.

Whateverthecauseofthisrapidcyclingofideas,theresult is that lessonsdevelopedover the

yearsthroughpost-projectevaluationsbecomemuchlessuseful,becausetheyareconsideredtheresultsofprojectdesignsnolongerconsid-eredvalid.

Thethirdfactorinourknowledge-managementgapiscausedbypersonnelchangesatthefieldlevel.Inmanydevelopmentagencies,peoplerotatethroughpositionsquiterapidly—espe-ciallyinfragileorpost-conflictregions.

TakeAfghanistan,wheredevelopmentagenciesmightpostsomeoneforoneyear.Becausethatpersonmaytakepersonalleavefromthecountryforatleastsomeofthatyear,theymightbeinstationonly40weeksoutoftheyear.Multiplythatacrossanoffice,andit’srarethatyouhaveafullcomplementofstaffatanyonetime.

Andthen,becausetheseareriskyenvironments,yougetcertaintypesofpeoplewhovolunteerfortheseassignments.Olderpeoplewithmoreexperience,whooftenhavefamilies,prefertoworkatheadquarters,soyougetyounger,lessexperiencedstaff.Thesearecomplexenviron-ments,wheretheexperienceandpoliticalnuanceareneeded.Afewpeopleareverycommittedandsimplygofromcountrytocountry,butformany,theygotothefieldearlyintheircareer,andthengobacktotheheadoffice.

Theresultisyoumayhaveaprojectconceptual-izedbyoneperson,designedbyasecond,com-missionedbyathird,monitoredbyafourth,andevaluatedbyafifth.Thequestionis:Howcanweeffectivelylearnfromthisprojectifthereislittlesenseofownershipandcontinuity?

Tobridgethisknowledge-managementgap,Ibelieveweneedtorampupin-projectmonitoring


andrealtimeimpactevaluation,sothatwecanlearnlessonstodayandthenfeedthoselessonsbackintotheproject.

Ofcourseeveryonewill say theyaredoingthis type ofmonitoring, but often it’s justanexerciseintickingabox.It’snotasexcit-ingorsexyasdoingthenewquasi-scientificthings likerandomcontrol trials,and long-termevaluation—bothofwhichhavearoletoplay—butitdoesgiveyoutheopportunitytocoursecorrect.

It’sabitliketheTitanic:Doyouwanttocoursecorrectandmisstheiceberg,ordoyouwanttosetupacomprehensivereviewofwhyithittheicebergandwhatlessonsyoucouldlearn?BotharevaluableapproachesbutIknowwhichshipIwouldratherbeon.

Andwiththeeconomiccrisisrightnowputtingpressureonaidbudgets,we’vegottomakesurethatoutofeverydollarwespend,wearegettingthebiggestdevelopmentimpact.Ifsomethingisnotworking,weneedtobeabletoidentifywhatisnotworking,takecorrectiveactionwhiletheprojectisongoingandgetbackontrack.

Somedonorsaredoingin-projectevaluation,anddoingitwell, includingtheMillenniumChallengeCorporationandtheU.K.’sDepart-mentforInternationalDevelopment.

There’s always a tendency to try andpolishthings,togetmethodologiesto100percent.Infact,whatisimportantistolearnfromwhatwe’realreadydoingsowecandoitbetter.

KPMG is walking that walk of knowledgemanagementthroughournewInternational

DevelopmentCenterofExcellence.BecauseKPMGisorganizedaroundclients,wehaveteamswithinKPMGworkingwithkeystake-holders in the development communityaroundtheworldincludingAusAID,theAsianDevelopmentBank,DfID,SIDA,theUnitedNations,theWorldBank,andmanyphilan-thropicorganizations.

Asyoucan imagine, thismeanswemaybeworkingonverysimilarissues,whichmakesglobalcollaborationandknowledgesharingevenmorecriticalasafactorinthesuccessofprojects.ThroughtheCenter,weaimtobringthoseinsightstogether,sowecanimprovedevel-opmentpractices.

It’sasmallstep,andwehavealongwaytogo.Butknowledgemanagementistooimportantnottogetright.Becauseattheendoftheday,developmentisnotaboutcomingupwithasmartmethodol-ogyorwritingacleverreport.It’saboutchangingpeople’slives,theycannotaffordtowaitforresultsandneithercanthedevelopmentcommunity.

This column was originally published by Devex Impact. Published here with the author’s permission.

Trevor Davies leads KPMG’s International Development Assistance Services Global Center of Excellence and works across the firm’s global network of member firms. He has more than 23 years of experience working with governments and develop-ment agencies.


Samer Hachem, Manager, Operations Evaluation Department, AfDB

My First encounter withthecon-ceptofcapacitydevelopmentdatesafewyearsback.Comingfromtheworldofmanagementconsulting,Iusedtoreen-gineer processes, diagnose organiza-tions,andmanagechange.Movingintotheworldofdevelopment,Iwastoldthatourjobwastobuildordevelopcapacity.Butwhatisthiscapacitywearesupposedtohelpdevelop?Whatkindofassistancecanmakeaninstitutionrunitselfsustainably?AndhowamI,aguywhohasbeendoingmonitoringandevaluation,supposedtohelp?

Obviously,Istartedwithdefinitions.Todayitisgenerallyacceptedthatcapacitydevel-opmenthastobeseenatthreedifferentlevels:individual,institutional,andcontext.Whiletheindividualleveldoesnotneedmuchexplanation,theothertwolevelsaremuchlessobvious.

Inabookwritteninthe1980s,GarethMorganproposeseightbasicmetaphorsonemightusetolookatorganizations.Thesemetaphorsproposedifferentlensestoanalyzeandunderstandhoworganizationsworkandevolve.Theyaremachine,organism,brain,

Developing Evaluation Capacity, but Which Capacity?

Samer Hachem

How harmoniously does the evaluation function operate with the other entities? How clear are the processes and structures surrounding evaluation? How well is knowledge packaged and delivered for decision making? How far does leadership accept this knowledge as a guide for action?


culture,politicalsystem,psychicprison,systemofchangeandflux,andinstrumentofdomination.1

Morgan’sbookisbasedonthepremisethatalmostallourthinkingaboutorganizationsisbasedononeoracombinationoftheeightmetaphors.Aswelookattheinstitutionsandorganizationsaroundusintheworldofdevelop-ment—developmentagencies,ministries,civilsocietyorganizations,and the like—wecanindeedseethesemetaphorsatplay.

Thinkofamachineforexample,andyoucanthinkstructuresandprocesses,andacontinuoussearchforthebestoutputatthelowestcost.Taketheorganism,andyouwillseeevolvingstructures,definedbyalevelofmaturityandinteractionstofunctionharmoniously.Takethebrain,andyouwillimagineorganizationsthathavemasteredknowledgeandhavelearnedhowtolearn.

ButMorgan’seightmetaphorsalsoprovideahelpfultooltothinkaboutcapacity.Capacityforamachine-typeorganizationisaboutexecut-ingthesameprocessconstantlywithefficiency,whilecapacityforthebrain-typeorganizationisaboutagilityandadaptation.

1 GarethMorgan, Images of Organisation (London, Sage Publications, 1986).

Evaluation,withitsdualpurposeofaccount-abilityandlearning,servestheorganizationasseenthroughitsmanyfacets,themachinebutalsothebrain,theculture,orthepoliticalsys-tem.Evaluatorsneedtounderstandthesemanyfacetstodelivertherightdiagnosis,butalso,andaboveall,toselecttherightpathtoimprovethevariouscapacitiesoftheorganization.

Tosupportevaluators,evaluationprocessesandinstitutionsneedtogrowandmatureovertime.Thenevaluationcandeliverfindingsandrecommendationsthatwillberelevant.Whatwillmakeadifferenceiswhetherthisknowl-edgethatevaluationgenerateshelpsshapeanenvironmentwhereknowledgeisvaluedasawayoflearningandevolving,inotherwords,wherethecontextofevaluationisconducivetoevaluation.

Howharmoniouslydoestheevaluationfunc-tionoperatewiththeotherentities?Howcleararetheprocessesandstructuressurroundingevaluation?Howwellisknowledgepackagedanddeliveredfordecisionmaking?Howfardoesleadershipacceptthisknowledgeasaguideforaction?Beyondindividualsandinstitutions,theseareultimatelythekeyquestionsevaluationcapacitydevelopmentneedstoanswer.

Samer Hachem is a manager in the Operations Evaluation Department of the African Development Bank. Prior to joining OPEV, he worked with the AfDB’s Quality Assurance and Results Department, where he focused on monitoring the Bank’s performance on aid effectiveness, and helped introduce the Development Effectiveness Review and a new results management system.

Samer started his career in management consulting, focusing on the financial services sector, and leading monitoring and evaluation in large-scale change management programs. He then joined UNDP HQ in NY, contributing to the roll out of new results-focused management policies and systems. He also worked as a consultant supporting West African countries’ efforts to evaluate their capacities in managing for development results and developing improvement plans.


Mohamed Manai, Manager, Operations Evaluation Department, AfDB

considerable progress has beenmadeinM&Eoverthelast20years.Duringthisperiod,theUnitedNationsDevelopmentProgramme(UNDP)hasplayedakeyroleinstrengtheningM&Ecapacityandbolsteringtheroleofotherfunders, includingthatoftheAfricanDevelopmentBank.

A1990regionalconference,heldundertheauspicesoftheOrganisationforEconomicCo-operationandDevelopment(OECD)(followedbyotherregionalconferencesinKualaLumpur,Malaysia,andinQuito,Ecuador)highlightedthenotableprogressmadeinM&E.Inparticular,discussionsunderscoredthatthetrendobserved—continuedtoestablishgoodgovernanceandeffectivemanagementofpublicresources—toinflu-encegovernmentsanddonorrelationshipstobuildanM&Eculture.Theconferencesalsohighlightedthatcountriesarenotallatthesamelevelofdevelopmentofnationalcapacityassessmentandthateachcountryhasfolloweditsownpath.Thiscallsforanapproachtailoredtothepolitical,administrative,andculturalcontextandthelevelofsocioeconomicdevelopmentofeachcountry.ItalsocallsforapreliminarydiagnosisofbothsupplyanddemandforM&Etoidentifywaysandmeansofimplementingtheinstitutionalization,professionalization,andimplementationofM&E,whetherinacentralized,decentralized,orparticipatoryframeworkorothermodel.

National Evaluation Capacity Development: Past and Future

Directions

Mohamed Manai, Manager, AfDB

Any national strategy for evaluation capacity building must emanate from the country itself through consultation with all stakeholders regarding national development plans or strategies for poverty alleviation and growth. Donors must coordinate their actions based on the resulting guidelines and national choices and work together with governments.


Therearethereforenobestpractices,buttherearegoodpractices.Allroutesarepossibleiftheyareorientedtowardimprovingthelegislativeframeworkforevaluation;theadministrativeframework,“constitutionalization,”orregula-toryframework;orensuringtheinvolvementandactiveparticipationofcivilsociety.

Anynational strategy for evaluationcapac-itybuildingmustemanatefromthecountryitselfthroughconsultationwithallstakehold-ersregardingnationaldevelopmentplansorstrategiesforpovertyalleviationandgrowth.Donorsmustcoordinatetheiractionsbasedontheresultingguidelinesandnationalchoicesandworktogetherwithgovernments.

A good example of this is the case of theDemocratic Republic of of Congo (DRC),wherethegovernmentsoughtsupportfromtheAfricanDevelopmentBank(AfDB),theWorldBank, and theUNDP tomodernizepublicadministration.Theseinstitutionswereengagedtoacttogetherinacoordinatedframe-work.TheUNDPcompleted thediagnosticstudyofnationalcapacity,whiletheAfDBandtheWorldBanksetupaframeworkforjointimplementationofresults-basedmanagementsystemassistance.Suchorganizationoftasksavoidsduplicationandoverlapofefforts.TheCLEAR(theRegionalCentersforLearningonEvaluationandResults) initiative,promotedbytheWorldBankandsupportedbytheAfDBandotherdonors,willbuildonthediagnosisofnationalcapacityassessmentscarriedoutbytheUNDP.Itwillnotreinventthewheel,but,instead,buildonexistingachievements.

Thefollowingtwoproposalscanhelpmovefor-wardtheNECDagenda:

1. Createaportalonthedevelopmentofnationalcapacityassessment.Thisportal couldbehostedontheUNDPwebsite(oranothersuit-ablesite),andwouldprovideaccesstotheexpe-riencesofcountriesthatwishtosharetheirbestpracticesandtheirdiagnosticmethodo-logicaltoolsfornationalcapacityassessment.Thisportalwouldalsoallowaccesstodeci-sionmakers,planners,researchers,academ-ics,practitioners,andnationalandregionalnetworks.Itwouldconstituteaframeworkforexchangeandinspirationthroughforumdis-cussionsonassessmentsofcapacitybuilding.ItwouldprovidelinkstositessuchasthoseoftheOECD,WorldBank,andUNDP.

2. Encouragedonorstoworktowardtheconver-gence,harmonization,andstrengtheningofpoliciesinordertoavoidduplicationorover-lap.TheharmonizationframeworkisavailablethroughtheapplicationoftheprinciplesoftheParisDeclarationandtheAccraAgendaforAction.Itistimelytorecallthatitispossibletodevelopcooperativesupportstrategies.TheCLEARinitiative,forexample,isamultidonorprogramthathascreatedregionalcentersinAsia,LatinAmerica,andAfrica(FrancophoneandAnglophone),butitmayevolveintoanexpandedprogramtomeetthelargevolumeofrequestsforevaluationcapacitybuilding.Thisprogramwillfundnotonlydiagnosticstudiesbutalsotradeactivities,travelstudies,training,andknowledgemanagement.Wehopethatthisprogramcaninterestotherdonorsandevolvequickly.

Insum,coordinationofdonorsinaneffectivepartnershipremainsinthehandsoftherecipi-entcountries,withsharedresponsibilityandmutualaccountabilityforimprovedpoliciesandefficiencyofdevelopmentaid.


Criticalsuccessfactorchecklistfordevelopinganationalevaluation (M&E) system

Critical success factors Key considerations for success

Drivers • WhatisdrivingthedemandforM&E?

• Whatarethebroadgoalsoftheexercise?

Uses • HowwillM&Einformationbeused,bywhomandforwhataudience(s)?

• Arethererealneedsforinformationthatarecurrentlynotbeingmet?

Leadership • Isleadershipsupportive?Isitleadingtheway?

• Istherea‘champion’fortheM&Eexercise?

Commitment • IstherecommitmenttolaunchanM&Eexerciseandtosustainit?

• WhatwillserveasthebasistoensurethatanM&Esystemgetsimple-mentedandsustained?

Resourcing • Wherewillthefinancialresourcestodevelopsystems,andhireandtrainskilledpersonnelcomefrom?

• Willorganizationsberequiredtointernallyreallocateorwilltheybegivennewmoney?

Accountability • WhowillbeaccountableforensuringthatanM&Esystemisafunc-tioningpartofthepublicsector?

• Haverolesandresponsibilitiesbeenfirmlyestablished?

Technicalcapacity

• Istherecapacity(datasystemsandinfrastructure)tocollectreliabledataandreportcredibleinformation?

• Isthereadequateanalyticalcapacity(skilledpersonnel)?

• Arethereexistinginstitutionsthatcouldserveascrediblepartners(e.g.nationalstatisticalagency,researchinstitutes)?

Infrastructure • Isthereapolicyandasetofstandardsinplacethatdescriberoles,responsibilitiesandexpectationsfortheoperationoftheM&EsystemandtheuseofM&Einformation?

• AretheorganizationsandunitsthatcollectandanalyseM&Einforma-tionstructuredandadequatelyresourced(budgetandhumanresourcecapacity)?

• IstheM&Einformationthatgetsresportedcredible,timelyandrespondingtothepriorityissues?


Critical success factors Key considerations for success

InfrastructuretouseM&Einformation

• Arethereformalpolicitesorrequirementsonhowperformancemoni-toringandevaluationinformationgetsusedbyorganizations?

• WhataretheincentiveswithinanorganizationforusingM&Einfor-mation(rewardsand/orsanctions)?

• Arethereformaloginformalvehicles/mechanisms/forumsforreport-ing,sharingortablingM&Einformation?

• Isresults-basedperformancefactoredintopersonnelassessments?

• Arecivilsociety,theprivatesectorandothersocialpartnersactivelyinvolvedinthenationalM&Esystem?

Oversight • Howwillthesystembemonitoredovertimetoensurethatitisfunc-tioningasexpectedandtothelevelexpected?

• DoesthenationalauditofficeplayanyroleinmonitoringtheM&Esystemandtheuseofperformanceinformationacrossgovernment?

• Isthereapolicycenter(e.g.withinacentralministry)tomonitorimplementation?

• WilltheperformanceoftheM&Esystembemeasuredandadjustedasnecessary?

ValuesandEthics

• Isthereaformalcodeofconductdescribingaccountabilitiesandexpectedbehaviourforofficialsinthepublicsector(e.g.transparency,accesstoinformation,fairandbalancedreporting,accountability?

• Isthiscodewellunderstoodandadheredtobyall?

• Is‘speakingtruthtopoverty’consideredappropriateinthepublicsec-torandnationally?

Sustainability • Whatrequirements/safeguardsaretheretoensurethattheM&Esystemwillbemadesustainable?

Source:Lahey(2006).National Evaluation Capacity Development: Practical tips on how to strengthen National Evaluation Systems,pps.14-15.

FONDS AFRICAIN DE DÉVELO

PPEMENT

AFRI

CAN D

EVELOPMENT FUND

BANQUE

AFRICAINE

DE DÉVELOPPEMENT

About the AfDB: The overarching objective of the African Development Bank Group is to foster sustainable economic development and social progress in its regional member countries (RMCs), thus contributing to poverty reduction. The Bank Group achieves this objective by mobilizing and allocating resources for investment in RMCs and providing policy advice and technical assistance to support development efforts.The mission of the Operations Evaluation Department is to enhance the development effectiveness of the AfDB in its regional member countries through independent and instrumental evaluations and partnerships for sharing knowledge

Operations Evaluation Department, African Development BankWebsite: http://operationsevaluation.afdb.org/Write to us: [email protected]

From Experience to Knowledge … From Knowledge to Action … From Action to Impact

Documents

Evaluation Matters - How can we Strengthen National ......• Strengthening National Evaluation Systems Samuel Kouakou, Côte d’Ivoire • The importance of Strengthening National