View
216
Download
0
Category
Preview:
Citation preview
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
1/21
WhatWeKnowAboutJointEvaluationsofHumanitarianActionLearningfromNGOExperiencesSection1of3:THEGUIDE
April2011
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
2/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 2
ContentsACKNOWLEDGEMENTS......................................................................................... 3
ABOUTTHISBOOKLET .......................................................................................... 4
THEGUIDE.............................................................................................................. 5
CHAPTER1:WHYDOAJOINTEVALUATION? .......................................................... 5TheBenefitsofaJointEvaluation ............................................................................................... 5
TheDownsidesofaJointEvaluation ........................................................................................... 6
Conclusion ................................................................................................................................ 7
CHAPTER2:JOINTEVALUATIONWHEN,WHOANDHOW? .................................... 8Whenwillittakeplace? ............................................................................................................. 8
Whowilltakepartinit? ............................................................................................................. 8
Isthereenoughtimeforajointevaluation? ................................................................................ 8
Howwillitbepaidfor? .............................................................................................................. 9
Howcanthejointevaluationbemostusefultovariousstakeholders? .......................................... 9
CHAPTER3:WHATTODOBEFORETHEEVALUATION ............................................ 10Choose
alead
agency
and
agree
on
roles .................................................................................. 10
Setupamanagementstructure................................................................................................ 10
Estimatecostsandduration ..................................................................................................... 11
Communicatewhattheevaluationisabout............................................................................... 12
Findacompetentadministrator/manager................................................................................. 12
Carefullypickevaluationteammembers................................................................................... 12
Chooseafewobjectivestocover.............................................................................................. 14
Agreeonevaluationstandardsandmethods ............................................................................. 14
Writeaninceptionreport......................................................................................................... 15
Managecommunicationswithinthecollaboration..................................................................... 15
Prepare,prepare,prepare!....................................................................................................... 15
CHAPTER4:WHATTODODURINGTHEEVALUATION ............................................ 16
Briefthe
team
upon
arrival ...................................................................................................... 16
Sharefindingsasyougo........................................................................................................... 16
Ensurefindingsarereportedwithsensitivity ............................................................................. 16
FinalizingtheEvaluationReport................................................................................................ 17
CHAPTER5:WHATTODOAFTERTHEEVALUATION............................................... 18Developbothcollectiveandindividualrolloutplans................................................................... 18
Emphasizepeeraccountability ................................................................................................. 18
CHAPTER6:JOINTEVALUATIONSINREALTIME .................................................... 19Preparefortheevaluationbeforetheemergencystarts............................................................. 19
Takeagoodenoughapproachtotheevaluation..................................................................... 19
Callonadditionalresources ..................................................................................................... 19
Considersomeotherjointreflectionprocess............................................................................. 20
Referencesand
Further
Reading.......................................................................... 21
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
3/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 3
ACKNOWLEDGEMENTSManypeoplehavesharedtheirvaluableexperienceandtimeinthecreationofthisbooklet.Special
thanksgotoallofthem,particularly,inGuatemala,CarlaAguilarofSavetheChildrenUS,HughAprileof
CatholicReliefServices,BorysChinchillaofMercyCorps,andJuanManuelGironDurinioftheECBProject;
inNiger,JasmineBatesandMariannaHensleyofCatholicReliefServices,andJuliannaWhite,ofCARE;in
Indonesia,AdhongRamadhanandJosephineWijiastutiofCatholicReliefServices,AgusBudiartoandEvi
EsalyKabanofSavetheChildren,HariningMardjukiandAnwarHadipriyantoofCARE,andRichardusIndra
GunawanandYacobusRuntuweneofWorldVisionInternational.SpecialthanksalsogotoJohnWilding,
PaulineWilson,JohnTelford,MauriceHersonofALNAP,JockBakerofCAREandGuySharrockofCatholic
ReliefServiceswhohavegivencriticalinputintothiswork.MalaikaWrightwastheauthorofthefirst
paper.
TheApril2011versionofthebookletwasupdatedbyKatyLovefromtheECBProject,LorettaIshidaof
CatholicReliefServices,JockBakerofCARE,HanaCroweofSavetheChildren,andKevinSavageofWorld
Vision.ThebookletwasrevisedbasedonfeedbackandreportsfromthosewhoparticipatedinECB
supportedjointevaluationsin2010inIndonesia,Haiti,theHornofAfrica,andNiger.Thesepeopleserved
asevaluationmanagersandcoordinators,teamleaders,teammembers,SteeringCommitteemembers,
ECBfieldfacilitators,andECBaccountabilityAdvisersinjointevaluations,including:PaulOHagan,Greg
Brady,YvesLaurentRegis,AngelaRouse,KatyLove,andJockBaker(Haiti);YenniSuryani,PaulineWilson,
LorettaIshida,andLeAnnHager(Indonesia);KevinSavage,CheleDeGruccio,JimAshman,andWynn
Flaten(HornofAfrica);andKadidaMambo(Niger).TheECBProjectthanksallwhocontributedtothis
work.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
4/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 4
ABOUTTHISBOOKLETThisbookletwaswrittentoshareknowledgegainedfromtheexperiencesofpeoplethathavebeen
involved
in
joint
evaluations
conducted
by
non
governmental
organizations
(NGOs).
It
mainly
profiles
the
workofNGOsinvolvedintheEmergencyCapacityBuildingProject(ECB),whichhasagoaltoimprovethe
speed,qualityandeffectivenesswithwhichthehumanitariancommunitysaveslives,improvesthe
welfare,andprotectstherightsofwomen,menandchildrenaffectedbyemergencies.
Thisbookletalsodrawsonthelessonsofmultiagencyevaluationsthatalreadyexistwithinthe
humanitariansector.Majorcontributionshavecome,inparticular,fromtheActiveLearningNetworkfor
AccountabilityandPerformanceinHumanitarianAction(ALNAP).
Wehopethatlearningfrompreviousexperiencescapturedherewillbeusefulforallthoseconsidering
leadingtheiragenciesthroughajointevaluation.Thelearningsharedhereistargetedatevaluation
practitioners,managers,andNGOscontemplatingajointevaluation.Additionally,wehopethatitwill
contributetoagrowingbodyofknowledgeontheseprocessesandshowthatwhiletherearemany
unansweredquestions
about
joint
evaluations,
there
is
alot
we
already
know.
Thisbookletiscomprisedofthreesections.Thefirstsection,TheGuide,canbereferredtoasahowtoforthosecloselyinvolvedinjointevaluations.Itprovidesaframeworkforthoseapproachingan
interagencyevaluation.Thesecondsection,TheStories,sharesseveralcasestudiesfromtheECBProjectsexperiences.Thethirdsection,TheTools,includesmanytemplatesandtoolsthatcanbeadaptedforevaluations,includingsampletermsofreferences,agreementdocuments,andchecklists.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
5/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 5
THEGUIDECHAPTER1:WHYDOAJOINTEVALUATION?Whyshouldanagencyconsidertakingpartinajointevaluationofanemergencyresponseprogram?
1
Afterall,jointevaluationsrequirecollaboration,collaborationmeansmoreworkandtime,andtimeisa
scarcecommodityinemergencyprograms.
Inrecentyears,severalNGOshavesoughttoanswerthisquestionwhiletakingpartinthejoint
evaluationsprofiledinthisbook.Whiletheresultshavebeenmixedandthelearningcurveshavebeen
steep,jointevaluationsconfermanybenefits.Whiletheevaluationsthemselveshaveyieldedinstructive
andusefulfindings,agencieshavealsobenefitedsignificantlyfromthequalityoftheinteractionsthat
tookplaceamongpeeragencies.Jointevaluationsoftenserveasforumsforongoinglearning,dialogue
andevenbegincollaboration.
Agenciesalsoinevitablylearnthattherearesomepitfallsintheprocessofconductingjointevaluations.
Thoughajointevaluationisnotsodifferentfromasingleagencyevaluation,therearesomemajor
differences,someofwhicharehighlightedbelowandaddressedthroughoutthisguide.
Aboveall,likeasingleagencyevaluation,ajointevaluationprovidesanopportunitytolearnfrompast
actionsoastoimprovefuturedecisionmaking.
Itshouldbenotedthatthisguidesetsouttheidealprocessesandstructuresforajointevaluation.Inan
emergencysetting,ofcourse,constraintsemergethatmaketheidealprocessachallengetoachieve.
Evaluators,therefore,mustbeflexibleandwillingtoadapttotherealitiesonthegroundinorderto
achievesomeifnotalloftheobjectivestheysetouttoachieve.2
TheBenefitsofaJointEvaluation1.
Seeing
the
Big
Picture
Oneevaluatorsaid,You[may]thinkyouvecoveredtheworldbutyouveonlycoveredone
villageinten.Emergencyresponsestypicallyinvolveseveralhumanitarianactors.Whenthe
responsesofmorethanoneactorareputsidebysideandexamined,theoverallpicture
becomesclearer,revealinghowfactorssuchasgeographiccoverage,sectorspecific
interventions,andcommunityinvolvementallfittogether.Jointevaluationsgofurthertowards
measuringimpactbylookingatthecollectiveeffortsofseveralactorstomeetbeneficiaryneeds
andtoidentifywhatgapsexist.
2. BuildingCoordinationandCollaborationtoImproveResponseGiventhescaleofdisastersandthedisproportionateamountofsufferingtheycause,agencies
workingalonearegenerallynotabletohavealargeimpact.Infact,agenciesthatcoordinate
responsesandworktogetherduringemergenciesarebetterabletomeettheneedsofdisaster
affectedpopulations.
By
comparing
agencies
responses
side
by
side,
joint
evaluations
are
better
abletopointoutareaswhereNGOscouldhaveactedinacomplementaryfashionandmake
1Themodelanddefinitionofjointevaluationsusedinthisbookletisanyevaluationthatlooksattheworkofmorethanoneagency.
Thisusuallymeansthatinadditiontomoreactorsbeinginvolved,thereisagreaterbreadthofprogrammingbeingexamined.2ReaderslookingforfurtherguidanceshouldreviewShoestringEvaluation:DesigningImpactEvaluationsUnderBudget,Time,and
DataConstraintsbyBamberger,Rugh,Church,andFort.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
6/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 6
recommendationsforhowtheycoulddosoinanticipationofthenextemergency.Evaluation
reportsrepeatedlyshowthatbettercoordinationwouldhaveledtoamoreeffectiveresponse.
Insomecaseswhereagenciesarealreadyworkingtogether,ajointevaluationcanbealogical
conclusiontoajointactionorresponse.InIndonesiaandinNigerin2010,agenciesagreedto
conduct
a
joint
evaluation
to
assess
the
impact
of
their
joint
activities.
3
Theprocessofcollaboratingontheevaluationitselfcanalsobeapowerfulwayofbuilding
relationshipsamongpartneragencystaffthatendureforthelongterm.IntheECBexperience,
someoftheserelationshipshaveledtoongoingactivitiesandeventheformationofanNGO
coordinationforum(seeNiger).InHaiti,thejointevaluationhelpedtobuildrelationshipsamong
nationalstaffandmanagers,servingasastartingpointforlongerterminteragency
collaboration.
3. WieldingWeightierConclusions,ImprovingPeerAccountabilityandTransparency
Jointevaluationscanbemoreauthoritativebecauseofthecombinedweightofthosebacking
them.As
such
evaluations
are
available
to
awider
audience,
there
is
likely
to
be
greater
pressure
toactupontherecommendations.Additionally,theyprovidealargerbodyofevidencefor
purposesofjointadvocacy.
Whenagenciesopenuptooneanotherbysharingweaknessesandstrengths,theyincrease
transparencyandmakeiteasierforthemtoholdonanotheraccountableforactinguponthe
recommendations.Transparencyiscriticalforagenciesinhumanitarianresponses,andsharing
thefindingsofevaluationsacrossagencieshelpstobecomemoretransparent.Infact,agency
peersmaypressuretheagencyactonrecommendationsfromanevaluation.
4. LearningfromandRelationshipBuildingwithPeersPartnersinajointevaluationhavearareopportunitytolearnabouteachothersprogramming
and
operations,
and
may
share
technical
knowledge
through
the
evaluation
process,
but
also
throughtheongoingrelationshipsthatareoftenestablished.Onepractitionernotedthat
workingwithstafffromotheragenciessometimesbringsnewperspectivesorevenchangesher
thinkingaboutaparticularissue.
Therelationshipbuildingthatoccursthroughajointevaluationallowsagencystafftoidentify
otheragenciesstrengthsandcapacities.Therelationships,foundedontrust,thatarebuilt
throughajointevaluation,mayresultinagencycooperationinthefuture.
TheDownsidesofaJointEvaluation1. MoreComplexity
Ittakes
time,
skill
and
patience
to
get
agencies
to
agree
to
do
ajoint
evaluation,
agree
on
amanageablelistofobjectives,diffuseanytensionsthatmayarise,ensurethatgroupdecision
makingprocessesareclearandrespected,allwhiledealingwithhiringandsupervisingan
evaluationteam,settingupinterviews,ensuringlogisticsareinplace,etc.Thisbecomeseven
harderduringanemergency.
3Seethe2010Indonesiajointevaluationreportformoreinformation,availableatwww.ecbproject.org/resources
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
7/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 7
Withoutaleadagencytotakeontheprimaryresponsibilityforthesetasks,andacommitted
steeringcommitteethatcanjointlyhandlestrategicdecisionmaking,ajointevaluationcanbe
frustratingandunsuccessful.
2. LessDepthOften
it
is
not
feasible
or
relevant
to
go
into
much
detail
on
any
particular
agencys
programs
as
wouldhappeninasingleagencyevaluation.Thereforemanyoftheevaluationquestionsof
interesttoeachagencymaynotgetanswered.3. MoreExpensive
Giventhenumberofactionsinvolved,jointevaluationscansometimesbemorecostlythan
singleagencyevaluations.Ifagenciesagreetosharethecostsoftheevaluation,however,
additionalcostsperagencywillbeminimal.
ConclusionJoint
evaluations
allow
NGOs
to
learn
from
multiple
perspectives
and
given
them
amore
complete
understandingofanemergencyresponse.Theyhelpusworktogethernowandinthefutureandleadto
relationshipsthatcanbeveryproductive. Forthesereasonstheycanbeenrichingexperiencesandhave
aprofoundimpactonthewaywedothingsasindividualagenciesandascollectives.Itisimportantto
havearealisticunderstandingofwhatcanandcannotbeaccomplishedbyajointevaluationbefore
conductingone.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
8/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 8
CHAPTER2:JOINTEVALUATIONWHEN,WHOANDHOW?Youmaywanttodoajointevaluationandhavegoodreasonstodoso.Butfirst,makesuretherewill
beenoughtimefortheevaluation,willingpartners,andhuman,financialandotherresourcestogetit
done.Thefollowingquestionsaremeanttohelpyoudeterminewhetherajointevaluationisfeasible.
Whenwillittakeplace?Evaluationscantakeplaceatdifferentpointsofaresponse(during,immediatelyafter,orseveralmonths
after).Thetimingdependsonwhattheagencieswanttogetoutoftheevaluation.Realtimeevaluations
duringaresponseprovideresultsthatcanimprovetheresponsegoingforward(seeChapter6).
Evaluationsconductedneartheendofanemergency,ontheotherhand,captureexperiencesand
learningwhileitisstillfresh.Evaluationsconductedwellafteranemergencyendscanstillbeusefuland
cancapturelongertermimpactofaresponse.
Oneimportantfactorwhencreatingatimelineforjointevaluationsistorememberthatworkingwith
multipleactorscanslowyoudown.Thereisrarelyaperfecttimetoconductajointevaluation,asall
agenciesarebusy.Therefore,especiallyforrealtimeevaluations,itisimportanttostartplanningasearly
aspossibleduringtheemergencyresponse.
Whowilltakepartinit?Approachotheragenciesthatmayalreadybeconsideringanevaluationforthesamehumanitarian
response.Consideragenciesthathavethesameoverallgoal(e.g.ensuringaffectedpopulationsareable
torecoverquicklyfromthedisaster),andthathavesimilartypesofprogramsingeographicareasthatare
closeenoughtogether.Identifytheappropriatepersontocontact,ideallysomeonewhoprovides
strategicdirectionforthecountryoffice.Explainwhatwillbegainedfromdoingthisevaluationjointly
(seeChapter1).Listentotheirviewsandnotethemdown.Dontbediscouragediftheyarenot
interested.Keeptalkingtootheragencies.
Whentalkingtootheragencies,findouthowtheyapproachevaluations.Dotheyconductthembecause
donorsrequirethem?Howdotheyusethefindings?Whatresourcesdotheydesignateforevaluations?
Takenoteofthistogetasenseofhoweachagencywillapproachtheevaluationandusethefindings.
Theiranswerswillalsohelpprepareyouforpotentialareasofconflict,suchaswillingnesstocontribute
stafftime. SeeTheTools,SuggestedTopicsforDiscussionwithProspectivePartners.Besuretheagencies
arewillingtocommitstafftimeandresourcestosupporttheevaluation.
Forajointeffort,andbecauseevaluationsmayrevealsensitiveissues,itsalsoimportanttobuildtrust
amongtheagencies.Todoso,agreeonthefocusoftheevaluationtogether,ratherthanapproaching
otherswithyourvisionandaskingiftheyareinterestedinjoiningin.Continuecollaboratingby
communicatingclearly,
being
transparent
with
information
and
intentions,
and
following
through
with
commitments.
Isthereenoughtimeforajointevaluation?Besuretoallocateenoughtimefortheevaluationteamtogetthejobdone.Unlessthelogisticsofgetting
toandfromfieldsitesisunusuallytimeconsuming,athirtyorfortydaycontractfortheleadevaluatoris
reasonable. Ensuretheevaluatorhasatleasttwodaysbeforeofficiallystartingtheevaluationtodo
preparatoryworksuchastoreviewdocuments,proposemethodology,andplanlogisticswiththe
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
9/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 9
agencies.Notensuringenoughtimefortheadditionalworkwillcompromisethequalityoftheevaluation.
Ensurethattimeisbuiltintoaccountformisseddeadlines,asmayoccurwhenmanyactorsareinvolved.
Howwillitbepaidfor?Jointevaluationsusuallytakemoretimetoconduct,andmayrequirearelativelylargeteam.Costs,
therefore,may
be
higher
than
for
single
agency
evaluations.
The
costs
can
be
spread
out
among
agencies,
andthisshouldbediscussedaspartoftheearliernegotiation.
Havearoughideaofwhattheevaluationmaycost.Themaincostsareforhiringconsultantsandsupport
staff.Comparethiswithwhatfundsmaybeavailableandwhatotheragenciesmaybewillingto
contribute,includingstafftime,lodging,andvehicles.Ifinsufficientfundsareavailablefortheevaluation,
considerajointpeerreviewtoreviewoneanothersprogramsandcometogethertodiscussfindings.
Properandrealisticbudgetingiscritical.
Donorsarelikelytobereceptivetojointevaluationsiftheybringaboutabetterunderstandingofthe
contextandtheoverallhumanitarianresponseandsomedonorscommissionjointevaluations
themselves.Therefore,ifabudgetfundedbyagivendonoralreadyaccountsforanevaluation,thedonor
maybeopentoredirectingthatactivityfromthesingleagencysevaluationtocontributetoajoint
evaluation.
Howcanthejointevaluationbemostusefultovariousstakeholders?Evaluationstakealotofresourcesandeffortandeveryonewantsthemtobeuseful.Jointevaluations
canbeusefultodifferentstakeholdersindifferentways.Inalargeemergency,agencystaffatregional
andgloballevelswilllikelybeinterestedinthefindings.
Talktopeopleattheheadofficelevelinthecountrywhere
theemergencyhappened,attheregionallevel,andat
headquarterslevel.Evenifthefindingsrefertoprograms
thathaveended,cantheybeusedtoinformother
programs,systemsandpolicieswithintheorganization?
Iftheproposalfortheevaluationcamefromheadquarters,dothoseinthefield,particularlycountryofficeleadership,
believethatthiswillbeausefulexerciseforthem?Ifnot,
theymaynotwanttoengage,andtheevaluationwillprove
hardtocarryout.Howwilltheyusethefindings?How
committedwilltheybetotheevaluation?Theirinterest
andengagementneedtobehightomakethisasuccessful
experience.
Thereshouldbeareasonablelevelofconfidencethatthe
findingswillbeusedbeforeproceedingwiththe
evaluation.Ifnot,theevaluationteamwillstruggletoachievetheobjectives.
TheideafortheECBsupportedGuatemala
evaluationcamefromheadquarters.Theteamin
GuatemalafeltthatthiswasanotherHQdriven
initiative,sotheirparticipationinsteering
committeemeetingswaslimited.Theagencieson
thegroundtriedtocustomizetheobjectives,butin
retrospectfelt
they
should
have
started
from
scratch.Thisnegativelyimpactedtheevaluation
processandthustheusageofthefindings.
Incontrast,theideafortheECBsupportedjoint
evaluationinJogyakartaalsocamefrom
headquarters.However,theparticipatingagencies
onthegroundtooktheleadondefiningtheir
objectives,withadvicefromheadquarters.This
helpedensurethepartnersweremoreincontrolof
theevaluationprocess.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
10/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 10
CHAPTER3:WHATTODOBEFORETHEEVALUATIONIfyouhavedecidedtopursueajointevaluation,herearesomethingstoconsider.
ChoosealeadagencyandagreeonrolesECBhashadthebestresultswhenoneagencyleadsthejointevaluationprocess.Thoughsomesharingof
responsibilityisdesirable,agenciesshoulddesignatethemajorityofthedaytodaymanagement
responsibilitiestotheleadagency.Anyoftheagenciesbeing
evaluatedcouldserveasthelead;whatmattersisthattheagencyis
capableofcarryingouttheresponsibilities.
Theleadagencyhiresandsupervisestheevaluationteam,coordinates
travellogistics,providesteammemberswithworkspaces,organizes
meetings,andgivesleadershipregardingthedefinitionofthe
objectives.Ultimately,itisthisagencythatisaccountableforensuring
thatthe
evaluation
takes
place.
Asteeringgroupmadeupofrepresentativesfromeachagencycan
cometogethertoagreeontherolesoftheleadagency,theroles
assignedtootherparticipatingagencies,andsharethemwithall
involvedpersons,staffandevaluators.(Formore,seesectionbelow
onManagementStructure.)
SetupamanagementstructureWhensettingupamanagementstructurefortheevaluationitsimportanttorecognizethatyouare
managingnotjustanevaluationbutacollaboration.Dontsuccumbtopressuretomakechoices
favorabletotheleadagency,steeringcommitteemembersorhighlevelsponsorsoftheevaluation(e.g.
wemust
have
x,
y,
and
zrepresented,
and
any
individuals
will
do).
Seek
out
individuals
for
the
steering
committeeandevaluationteamwhoarecommittedtoasuccessfuloutcome,eveniftheyarenot
conventionalchoices.Wherethereisaneedforagencyrepresentation,createspacefortheseindividuals
insomehighprofile,butlesscriticalfunction.
Ajointevaluationmanagementstructurecouldlooksomethinglikethis:
Asteeringcommittee. Thisgroupwillberesponsibleforstrategicdecisionmakingveryearlyonregardingobjectives,timing,andresourceallocation,includingstaffandfunding.Thesteering
committeewillalsobeactiveinreviewinganddebatingthefindingsandactingupontheir
implicationswithintheiragenciesandbeyond.Itisnormallychairedbytheleadagencyandhas
representationfromeachoftheparticipatingagencies.Thecommitteewouldideallybekepttoa
maximumoffive,makingoversightanddecisionmakingmorefocusedandachievablein
reasonableamounts
of
time.
This,
however,
supposes
that
agencies
involved
are
willing
to
delegatestafftoacommittee.
Theidealsteeringcommitteememberisseniorenoughtospeakonbehalfofhis/heragencyand
hastheauthoritytomakedecisions.Thisindividualmusthaveagoodknowledgeofhisorher
organizationsemergencyprogramsandongoingdevelopmentwork.Inaddition,heorshe
shouldbeabletothinkstrategically,andknowenoughaboutevaluationstoadviseonthe
evaluationmethodstobeusedandonthefieldlocationstobecovered. Theseindividualswill
alsobethosemostlikelytofollowuponrelevantrecommendations.
Aleadagencythatplaysits
rolewellcanmakeamajor
differenceintheprocess.The
headevaluatorinECBsNiger
evaluationfoundthelead
agencysorganizationofthe
evaluationprocessand
logisticstobethemosthelpful
thingto
him
in
carrying
out
his
work.Itwasoneissuewe
didnthavetothinkabout;it
wassowellorganized,he
noted.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
11/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 11
Steeringcommitteemembersshouldagreeonanddocumentclearprocessesandstandardsof
efficiency,transparencyandaccountabilityregardingrolesandresponsibilities.Theyshould
agreeonhowdecisionmakingwouldwork,howtoresolvedisagreementswithinthesteering
committee,andhowtoshareinformation. Steeringcommitteemembersshouldbethesame
throughout
the
entirety
of
the
evaluation.
When
members
rotate
on
and
off
the
committee,
decisionsandguidancemaychange,whichwillcomplicatemattersfortheevaluationteam.
Otheragreementsconcernthereportformat,theuseofagencylogos,theownershipofthe
productsoftheevaluation(i.e.intellectualcopyright),howagencieswillusethefindingsandif
theywillholdoneanotheraccountable,etc.
Achairperson.Thispersonisbasedattheleadagencyfortheevaluationandisamemberandchairofthesteeringcommittee.Heorshehasmostofthestrategicdecisionmaking,operational
andcollaborationresponsibilitiesoftheevaluation.Thisindividualusuallyassumestheroleof
evaluationmanagerandisthedirectreportinglinefortheteamleader.Thechairpersonshould
managethebudgetandtrackexpenses.
A
manager
or
administrator.
This
person
must
ideally
be
based
in
the
lead
agency
with
a
certain
percentageofhisorhertimededicatedtotheevaluation.Seethefollowingsectionformore
detailsonthemanagersresponsibilities.Heorshecouldalsositonthesteeringcommitteebut
withoutvotingrights.
Theevaluationteam. Theteamistypicallycomposedofoneortwoindependentconsultants,andarepresentativefromeachofthepartneragencies.Thisteamisaccountabletothesteering
committee,particularlythecommitteechair.
Therearevariationsonthisstructure,ofcourse.Mostevaluationsalsohavehigherlevelsponsors
thatmayalsoformasuperstructure.Sectorexpertsmayalsobeneededontheevaluationteam.
SeeTheToolsforaSampleAgreementsDocument.
EstimatecostsanddurationBasedonthedraftitinerary,thesteeringcommitteeshouldagreeonadraftbudgetandcostsharing
arrangements.Typicallyagenciesshareconsultantcostsequallyandprovidefundingforthestaffmember
theyappointtojointheevaluationteam.Thinkthroughfundingimplicationsforallaspectsoftheprocess
andhowlongeachactivitywilltake.Forexample,ensurefundsforgoodqualityediting,formatting,and
presentation,asthesecanmakeasignificantdifferenceinhowwidelythereportisread.Berealistic
aboutthetimeitwilltaketheevaluationteamtogetthejobdone.Atleast30daysorevenfortydaysare
recommendedfortheteamleader.Theconsultantwilllikelybethelargestcost,butitisessentialto
budgetfor,ashis/hertaskswillinclude:
Reviewdocuments,preparemethodology,andcorrespondwiththesteeringcommitteepriorto
theevaluation.
Conductfieldvisitstoatleastthreesitesforeachoftheagencies.
Interviewagencystaff.
Interviewotherstakeholders.
Presentthefindingstostakeholdersincountry.
Prepareadraftofthereport. Incorporateeditsandcommentsonthereportfrommultipleactors.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
12/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 12
CommunicatewhattheevaluationisaboutAjointevaluationisanewsworthyevent,butnoteveryonewillunderstanditspurpose.Makesurepeople
insideandoutsidetheparticipatingagencies,includingbeneficiaries,areawareoftheevaluationsothat
theywillbemorelikelytoreviewandmakeuseofthefindings.Draftaonepageinformationalsheet
abouttheevaluationforwidespreadsharing,andespeciallywithcountryofficestaffwhoneedtobe
awareof
the
evaluation
(though
they
would
ideally
be
engaged
throughout
the
process).
Itisparticularlyimportanttohavepreparatorydiscussionswithbeneficiarycommunitiestoensurethey
understandthepurposeoftheevaluationandtheyagreetoparticipateintheevaluation.Theyshould
understandthatevaluatorsdonothaveanyassistancetogive.4InHaiti,theevaluationteamtrained30
nationalstafffromtheparticipatingagencieswhospokeHaitianCreoletoberesponsibletoengage
beneficiaries.Theydidsobyconductingfocusgroupdiscussions,askingopenendedquestionsto
understandpeoplesexperiencewiththeemergencyresponse.5
Findacompetentadministrator/managerConsiderhiringsomeonewhocanspendasignificantamountoftime(50100%)focusedonthe
evaluation,especiallyinthemonthortwoleadinguptoit.Thispersonmaybeanadministrator,but
shouldalso
be
supervised
by
asenior
person
who
can
advise
on
strategic
issues.
Asuperbadministratorcanmakeamajordifferenceinthesuccessofanyevaluationbutparticularlya
jointevaluation.Ideally,anationalstaffpersonshouldbehiredorsecondedfromoneoftheagencies.He
orshewillberesponsibleformeetingthelogisticalandadministrativeneedsofboththesteering
committeeandtheevaluationteam.Asampletasklistforthispersoncouldlooklikethis:
Organizetherecruitmentfortheindependentconsultant(s).
Draftandprocesscontractwithconsultant(s).
Arrangeschedulesandmanagethecalendar.
Arrangelogisticalarrangementsfortravelinginthefield.
Coordinateinformationexchangebetweentheagenciesandtheevaluationteam,such
ascollectingtherelevantbackgrounddocumentsfortheevaluationteam.
Arrangemeetingsbothwiththeparticipatingagenciesandwithoutsideactors.
Helpdocumentwhoisresponsibleforwhatandsharethiswithallparties.
Agreeonnormsforperdiemandotherpolicies.Typically,eachagencyfollowstheir
ownandthecoordinatingonehirestheexternalsandusestheirperdiems.
Meetwiththeevaluationteam.
Thesteeringcommitteeorchairpersoncouldappointtheevaluationadministratorormanager.Ideally,
thesteeringcommitteewilldefinetheauthorityleveloftheadministrator,whoheorshewillreportto,
andwhatlevelofauthorityheorshewillhavetomakedecisions.Itshouldbemadecleartheamountof
timeanadministratorwillprovidetosupporttheevaluationteam.
CarefullypickevaluationteammembersSelecttherightteam.Inadditiontothetechnicalskillstheyneedtoconducttheevaluation,team
memberswillalsohavetobegoodatbalancingtheneedsofmultipleclientswithsensitivity.Whiletheir
rolesshouldbemadeclearbeforetheevaluation,experiencehasshownthattheywillneedtobeflexible
4SeeTool9intheGoodEnoughGuidetoImpactMeasurementandAccountabilityinEmergencies
5SeetheCARESavetheChildrenjointevaluationreportatwww.ecbproject.org
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
13/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 13
oncetheevaluationbegins.Theteamshouldnotbetoolargethatitisdifficulttomanage.Threetofour
teammembersisusuallysufficient. Atypicalteammaybecomposedof:
Anindependentconsultant/teamleader.Thispersonknowsalotaboutevaluationandalsohasstrongmanagementandleadershipskills,theabilitytostaycalmunderpressureandtobe
adaptive
in
the
face
of
the
unexpected.
A
joint
evaluation
team
leader
also
needs
the
ability
to
dealwithmultiplelayersofmanagementandbalancevariousexpectations,andthusmusthave
strongdiplomacyandcommunicationskillsbothwrittenandverbal.Theteamleadershould
alsohavehadpreviousexperienceasateamleader,sincethisisitselfaspecialskill.Team
leaderswiththeseskillsetsaresometimeshardtofind,anditiscriticalthatthereisabudgetline
topayforthem.
Thoughitsnotalwayspossibletorecruitateamleaderwhohaspreviouslyledjointevaluations,
confirmthatheorshehasexperienceinimpactanalysisinemergencies,asheorshewillneedto
understandhowthevarioussetsofdatacometogethertoformabiggerpicture.Notethat
consultantsoftencomewiththeirownideasandmethodologies,andtheywillneedguidance
andparametersfromthesteeringcommittee.
Anationalconsultant.Thenationalconsultantprovidescriticalguidanceonthepolitical,social,andculturalcontextoftheemergencytotheteam,especiallytotheteamleader,whoisoftenan
expatriate.Havingsuchapersononhandforajointevaluationcanhelpinnetworkingwith
nationalstakeholdersandensurethatknowledgeisquicklytransferredtotheevaluationteam
aboutkeyactorsandeventsandcanminimizesomeofthecomplexityofthedataandfactorsto
beanalyzed.
Asectorspecialist. Ajointevaluationwillchallengetheteamofevaluatorstoaddressthewiderangeofprogramareasbeingcoveredwhilealsofocusingonselectedkeyandpriorityaspects,
especiallyaseachagencymayhaveuniqueinterests.Ifagenciesneedmoreindepth
examinationofaparticulartypeofprogram,theyshouldconsiderbringingasectorspecialistto
theteam,freeingupothermemberstofocusontheoverallpicture.
Agencyteammembers.Eachagencytypicallyappointsonerepresentativetotheevaluationteam.Theseindividualsarenotactingonbehalfoftheiragencybutrathermustbeimpartial
evaluators.Theskillsetsofthesepeople,forexampletheirexpertiseincertainsectoralareas,
languageandfacilitationskills,andevaluationexperienceareveryimportanttotheoverall
successoftheteam.Ensuringthatagencyandevencountryofficestaffarerepresentedon
theevaluationteamwillincreaseownershipoftheevaluationfindings.
Itmaybehardforagencyteammemberstobeavailableforthefulllengthoftheevaluation,but
experiencehasshownthatcontinuityisimportanttoevaluationqualityandthelearning
experienceisalsogreatlyenhanced.Agencymanagersshouldthereforemakeeveryeffortto
ensurefullparticipationofagencystaffontheevaluationteam.
Giventhe
importance
of
getting
competent
team
members,
its
important
to
start
the
recruitment
processearly.Goodindependentconsultantsnationalandinternationalalikeareoftenbookedfor
weeksorevenmonthsinadvance.
OncethesteeringcommitteehasfinalizedtheTermsofReferenceforteammembersandtheskillsthey
want,agenciesshouldconsiderrequestinghelpfromtheirheadorregionalofficesinrecruitingtheteam,
suchasdoingtheinitialadvertisingandscreeningandthensendingashortlistofcandidatestothelead
agency.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
14/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 14
Bythetimehiringbegins,theobjectivesfortheevaluationshouldhavebeenwellenoughdefinedthatthe
steeringcommitteeisclearonwhatprofilesareneededontheteamandthus,whotohire.
SeeTheToolsforSampleTermsofReferenceforEvaluationTeamMembers.
ChooseafewobjectivestocoverTheparticipatingagenciesmayhavedifferentinterestareastheydliketocoverinajointevaluation.But
itisnotpracticaltoaddresstoomanyobjectivesinajointevaluationasthereisalreadymorecontentto
cover.Ideally,thereshouldbenomorethanthreeorfourobjectiveswithinthetermsofreference,and
thescopeshouldbeasnarrowaspossible.Forexample:
Howwelldidthevariousagenciescoordinatetheirresponses?
Howappropriatewastheintervention?
Howtimelywastheintervention?
Howwelldidtheresponseassistpeopleinrecoveringfromthedisaster?
Objectives
that
concern
the
overall
impact
of
the
response
are
usually
best
for
a
joint
evaluation.
Objectivesofuniqueconcerntooneortwoparticipatingagencies,suchasissuesofoperationalefficiency,
arenotgenerallyappropriate.Inareaswheremoredepthisneeded,hireanadditionalteammemberto
focusspecificallyonaparticulartypeofprogrammingorissue.
Doaskforinputonthescopefromstaffatdifferentlevelsofeachagencywhomyouexpecttousethe
evaluationfindings.Atthesametime,itiswisenottoconsulttoowidely,asyouwillruntheriskofadding
toomanyobjectivesandanunrealisticscopefortheevaluation.
Objectivesshouldbeagreeduponbeforetheevaluationteamishired.Infact,considerbringinginan
externalfacilitatortonegotiatethescopeoftheevaluationaheadoftime.Oncetheleadevaluatorjoins,
heorsheshouldhavethechancetotelltheparticipatingagencieswhatisfeasibleandrealistic. Itis
criticaltofindabalancebetweenwhatagencieswantandwhattheleadevaluatorbelievesispossible.
SeeTheToolsforaTermsofReferenceTemplate.
AgreeonevaluationstandardsandmethodsJointevaluationsshouldincludeadocumentreview,keyinformantinterviewsandfocusgroupdiscussions
withstaffandbeneficiarygroups.
Theteamleaderwillbuildanapproachtoexamineeachagencysworkwithenoughrigortoinspire
confidenceinthefindings,butnotdetracttoomuchfromafocusontheoverallimpactoftheagencies
response.However,thesteeringcommitteeisexpectedtoadvisethisprocessandalsocommunicatethe
criteriatheywilluseforvillageandbeneficiaryselectionfortheinterviews.
Inadditiontomorelocationsforfieldvisits,forjointevaluations,theremayalsoneedtobemore
interviewswithotheractors,suchasUNagencies,representativesfromcivilsociety,nationalandlocal
partners,andgovernmentofficials.
Certainindicatorswillbenonnegotiabletobeinlinewithacceptedinternationalstandards,suchasthe
OECD/DACstandardsforevaluation.6 Spherestandardsareanotherkeypointofreferencewhichshould
6http://www.alnap.org/resources/guides/evaluation/ehadac.aspx
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
15/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 15
beassessedduringanevaluation,notonlyfortechnicalsectors,butalsocorestandardsofparticipation.
Onesetofstandardsshouldbeusedforconsistencyinmeasuringperformance. Beclearonwhat
organizationalminimumstandardsare.ReferencetheKeyElementsofAccountabilityontheECBProject
website.
WriteaninceptionreportTheevaluationteamshoulddevelopaninceptionreportonthetermsofreferenceandwithaworkplan.Thisreport,writtenbytheteamleader,willensureexpectationsareagreedonbythesteeringcommittee
andtheteamitself.Italsoallowsfortheevaluationteamleadertodialoguewiththesteeringcommittee
aboutwhatisrealisticandfeasible,givenavailabilityofstaff,budget,anddeadlines.
ManagecommunicationswithinthecollaborationAgenciesconductingajointevaluationneedclearagreementsaroundcommunication.Facetoface
meetingsarecriticaltomakesureunderstandingsareclearandtobuildcohesion.Decidinghowtostore
keydocumentsisalsoveryimportant.Onesolutionistosetupasimplewebpagetouploaddocuments,
contactlists,schedulesandotheressentialinformation.It
is
also
important
to
have
regular
opportunities
along
the
way
for
the
evaluation
team
to
discuss
any
concernswithsteeringcommitteemembers.Forexample,earlyintheprocess,theteamcangive
feedbackastohowwelltheevaluationmethodsareworkingandcheckwiththesteeringcommittee
whethertheseshouldbemodified.Ifthesteeringcommitteeisengaged,theevaluationwillbemuch
morelikelytosucceed.
Itisalsoimportanttoagreeinadvanceonprinciplesoftransparency withevaluationresults,including
communicatingresultsinatransparentwaytobeneficiarycommunities(whichcouldbeintheformofa
discussionorroundtable). Tryingtocoveruporhideevaluationresultsisnotonlyagainstprinciplesof
accountability,butunderminesorganizationallearningandcanoftenbackfire.
Prepare,prepare,prepare!Our
experience
has
shown
that
agood
amount
of
work
can
be
undertaken
even
before
the
evaluation
teamarrives.OncetheTermsofReferencefortheevaluationhasbeenestablished,alistofkeyinformant
interviewscanbedetermined,meetingsestablished,focalpointsready,andpreparatorydocumentscan
beemailedtotheevaluator.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
16/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 16
CHAPTER4:WHATTODODURINGTHEEVALUATIONIfrealisticobjectives,managementstructure,andacompetentteamhavebeenchosenandestablished,theevaluationshouldbeeasiertomanage.Theteamwillstillneedgoodlogisticalsupport
andguidance.Hereissomeadditionalguidanceonconductingtheevaluation.
BrieftheteamuponarrivalEnsurethattheteamhasachancetodiscusstheTermsofReferencewitheachofthesteeringcommittee
members.TheleadevaluatorshouldalsogoovertheTermsofReferencewiththesteeringcommitteeas
agroup.
Theadministrator/managerorsteeringcommitteechaircanbriefteammembersonrolesand
responsibilitieswithintheevaluationstructure.Theteamwillneedtobebriefednotonlyonthelogistics
andtheprocessoftheevaluation,butalsoontheresponseprogramswhicharetobeevaluated.The
team
will
need
to
be
clear
on
how
the
evaluation
is
run,
the
role
of
the
lead
agency
and
the
other
agencies,towhomtheteamreports,wheretheywillgetlogisticalsupport,andhowtheywillmaintain
independence.Whenthesearenotclear,confusionabounds,andtheteamwillstruggletoachieve
objectives.Anticipatetheextraconsultationtimeneededwhenestimatinghowmuchtimetheteamwill
needfortheevaluation.
SharefindingsasyougoTheteamleadersshouldalsoensurethatthesteeringcommitteeandthestakeholders(asmentioned
below)receiveregularupdatesthroughouttheprocess.Ifthesteeringcommitteeandstakeholdersare
wellbriefedabouttheprogressandinitialfindingsoftheevaluation,therewillbenosurprisesattheend.
Dailydebriefsamongtheevaluationteamdrawsoutpreliminaryfindingswhichtheteamleadercanuse
toprovideregularupdates.
EnsurefindingsarereportedwithsensitivityReceivingandreviewingthefindingsofajointevaluationcanbeanexcitingtimefortheagenciesbutalso
atimeofapprehension.Theleadevaluatorshouldpresentfindingsinawaythatwillnotmakeanyagency
feelinferiororunfairlycomparedwithothers.Agencieswillalsoinevitablylookformentionsof
themselvesandjudgewhethertheythinkthefindingsarefair.Findingsthatarecriticalinnatureshould
bephrasedinaconstructiveway,supportedbyreasonableevidenceandbalancedwithpositivefeedback.
Inadditiontothemainreport,theevaluatorcouldalsocreateshortindividualreportsforeachofthe
agencies.Inpractice,however,thismaynotbeworththeadditionaleffortsincejointevaluationstendto
bebetteratlookingattheoverallresponseandcoordinationbetweenagencies(i.e.fromabeneficiary
perspective)thanlookingatindividualagencyoperationsindetail.
Anotherapproachisfortheevaluationteamtodoapreliminaryanalysisthatcompilesandgroups
findings.Throughaworkshopormeeting,theteamcanfacilitatestafffromtheparticipatingagencies(especiallythosewhowillusethefindings)tocollectivelydrawconclusionsandrecommendations.With
thistypeofparticipation,agenciesaremorelikelytoaccepttheconclusionsandfeelresponsiblefor
actingontherecommendations.
Ultimately,whenjointevaluationsarewellplannedandagenciesandtheteamcommunicatethroughout
theprocess,agenciesarelesslikelytotakeissuewiththeresults.Afocusonlearningmakeseventhe
leastflatteringfindingsmorepalatablebecausetheycanbeinstructive.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
17/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 17
Oncetheagencieshavehadachancetodiscussanddebatethefindings,theyshoulddiscussthemwitha
broadergroupofstakeholders,especiallythosethatwereconsultedduringtheevaluationprocess,such
asUNagencies,beneficiaries,localNGOs,andlocalgovernment.Onewaytodothisistoholdaninter
agencyvalidationworkshopwherestakeholdersaregivenanopportunitytoconfirmordisputethemajor
findings
and
recommendations.
FinalizingtheEvaluationReportTheevaluationreportshouldbeeasytoreadandrelativelyshortnomorethan30pages.Itisimportant
tofocusonthatwhichhasgonewell,andgoodpracticeshouldbehighlightedinthereport.
Assumingstakeholdershavebeenbriefedthroughouttheevaluation,thefindingsandrecommendations
intheevaluationshouldnotcomeasasurprise.Doanticipate,however,thatstakeholderswillnotagree
withallfindingsandthesteeringcommitteeshouldbepreparedtoaddressthis.
Itiscriticaltosetoutaperiodtoreceivefeedbackonthedraftofthereport.Beclearandrealisticabout
thetimelineforthisperiod.Itneedstobelongenoughtoallowtherightpeopletoprovidefeedback,but
notsolongthatthefindingsarenolongerrelevantbythetimethereportiscompleted.Afterensuring
thatallofthepeoplewhoneedtogivefeedbackareinformedofthescheduleinadvance,twotofour
weeksisareasonabletimeinwhichtoallowpeopletosubmitfeedback.
Theevaluationteamleaderisultimatelyresponsibletomakethedecisionsaboutwhichfeedbackis
incorporatedandwhichisnot.Ifthereisenoughdisagreementaboutcertainfindingsorconclusions,
thesecanbeaddressedinamanagementresponsethatisannexedtothefinalreport.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
18/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 18
CHAPTER5:WHATTODOAFTERTHEEVALUATIONWiththeevaluationworkdoneandthefindingsdetailed,onepartoftheprocesscomestoaclose.But
inotherways,therealworkisjustbeginning.Hereissomeguidanceformakingthemostofthe
completedevaluation.
DevelopbothcollectiveandindividualrolloutplansBecausejointevaluationshaverelevancetoawiderangeofactors,agenciesshouldsharethereportwith
humanitarianbodiesandsuchnetworksastheActiveLearningNetworkforAccountabilityand
PerformanceinHumanitarianAction(ALNAP),inadditiontoheadquarters.Sharingthereportfromajoint
evaluationreportwidelydemonstratestransparencyandacommitmenttocontributetolearningwithin
thebroaderhumanitariansector.
Theagenciesmaywanttodevelopsimplecollectiveandindividualcommunications plansincluding
distributionlistsforthereportandsmallactionplanningmeetingstodiscussandpresenttheimplications
ofthefindings.
EmphasizepeeraccountabilityWithjointevaluations,agencieshavetheopportunitytoholdoneanotheraccountableforprogresson
recommendations.Theymaychoosetoworkonsomerecommendationstogether.Theymayagree
beforehandtoholdafollowupworkshopinsixmonthsorayearstime.Atsuchatime,theycouldthen
discusshowthefindingswereshared,whatprogresswasmadeandwhatwastheoutcomeofanyactions
taken.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
19/21
CHAPTER6:JOINTEVALUATIONSINREALTIMEConductedwhiletheemergencyresponseisstillongoing,realtimeevaluations(RTEs)arevaluable
toolsforrectifyingproblemsandmakingimprovementsinprogrammes.However,ajointRTEcanbe
especiallychallenging.
Here
are
some
things
to
consider
iffacing
the
decision
of
conducting
ajoint
evaluationinrealtime.
PreparefortheevaluationbeforetheemergencystartsIncasesofslowonsetemergencies,theremaybetimetoplanmonthsinadvance.Evenwithrapidonset
emergencies,preparednessispossible.Youcanjointlyoutlinegenericplansfortheevaluationwhichcan
beturnedintoactualplansinthefaceofadisaster.Theseplansshouldincludeasmanyoftheaspectsas
possiblethatareoutlinedinthisbookletonhowtoorganiseajointevaluation,including,crucially,the
designationoffocalpoints.Thesefocalpointsareonstandbyandwillhavetheresponsibilityofgettingan
RTEprocessstarted,conveningthevariousactors,etc.andwhowillideallyremainaspointpersons
duringtheprocess.
TakeagoodenoughapproachtotheevaluationYoumayhavetotakesomeshortcutsanduseagoodenoughapproach.Goodenoughdoesnotmean
secondbest:itmeansinanemergencyresponse,adoptingquickandsimplesolutionsmaybetheonly
practicalpossibility.Whenthesituationchanges,youshouldaimtoreviewyourchosensolutionand
amendyourapproachaccordingly.
Forexample,youcansimplifymanagerialstructures.Withindaysofthepartiesagreeingtodoajoint
evaluation,youmayagreetoestablishasmall,rapidlyorganizedmanagerialstructure(whichcan
subsequentlytransitiontoamorerobustoneatalaterstage).Duringthefirstweek,forinstance,that
groupcouldlookatwhatisminimallynecessary,andcreateapracticalquickanddirtytermsof
reference.Themanagementcommitteecoulddelegatemuchofthedaytodaymanagementtooneor
twokeyactors,andthusspendlesstimeongroupdecisionmakingandconsensusbuilding.
Eachparticipatingagencywouldbetrustedtocarryoutthetasksassignedtotheminaccordancewith
predeterminedplansandstandards.Oncetheprocesshasbeenstartedandtheevaluationisinmotion,
agenciescanthengraduallybuildintighterqualitycontrolmechanisms,morefocusedtermsofreference,
andamoreinclusiveprocess(e.g.alargermanagementgroup).
Suchagoodenoughapproachisnotanidealevaluativeprocess,butlikelyrelevantforRTEsbecause
agenciesareparticularlybusywiththeimplementationofaresponse.However,certainaspectsofthe
jointevaluationshouldnotbesubjectedtoshortcuts.Theseincludeethicalstandards,suchasthe
confidentialityandindependenceoftheevaluativeprocess.
CallonadditionalresourcesParticipating
agencies
could
consider
calling
on
additional
internal
support.
A
staff
member
could
be
secondedtoacountryofficeforsomeweekstohelpwiththejointRTE.Unlikecountryofficestaff,who
wouldpresumablybepreoccupiedwiththeemergencyresponse,thispersonwouldhavetimetofocuson
theevaluation.Heorshecoulddoaninitialscoping,astakeholderanalysisandholdameetingcollectively
orindividuallywithpartnerstogettheirviews.Heorshecouldalsoassistwithpracticalpreparationsfor
theteam,includingsettingupfieldvisits.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
20/21
WhatWeKnowAboutJointEvaluations TheECBProject info@ecbproject.org April2011(v6) 20
ConsidersomeotherjointreflectionprocessIfajointRTEisnotrealistic,considerotherlearningprocesseslikeajointafteractionrevieworapeer
review.Agenciescandoquickassessmentsoftheirwork(seeImpactMeasurementandAccountabilityin
Emergencies:TheGoodEnoughGuide)andgettogetherforashortmeeting/workshop,orinvitean
experiencedexperttoprovideadviceonhowtheoperationmaybeadapted.
8/22/2019 Guidelines for Conducting Joint Evaluations of Humanitarian Action
21/21
ReferencesandFurtherReadingJointEvaluationsGuidanceforManagingJointEvaluations.DACEvaluationSeries,OECD2006.
http://www.oecd.org/dataoecd/29/28/37512030.pdf
JointEvaluations:RecentExperiences,LessonsLearned,andOptionsfortheFuture.DACEvaluation
NetworkWorkingPaper,OECD,2005.
LessonsAboutMultiAgencyEvaluations:AsianTsunamiEvaluationCoalition.http://www.tsunami
evaluation.org/NR/rdonlyres/9DBB5423E2EF43ABB6D2
2F5237342949/0/tec_lessonslearned_ver2_march06_final.pdf
GeneralEvaluationsUSAIDCenterforDevelopmentInformationandEvaluation,PerformanceMonitoringandEvaluationTIPS
series.
http://evalweb.usaid.gov/resources/tipsseries.cfm
WesternMichiganUniversity,EvaluationCenter.EvaluationChecklists
http://www.wmich.edu/evalctr/checklists/checklistmenu.htm#models
ShoestringEvaluation:DesigningImpactEvaluationsUnderBudget,Time,andDataConstraints. M.
Bamberger,
J.
Rugh,
M.
Church,
and
L.
Fort,
The
American
Journal
of
Evaluation,
2004.
UtilizationFocusedEvaluationChecklist.MichaelQuinnPatton
http://www.wmich.edu/evalctr/checklists/ufe.pdf
Recommended