COLUMBIA LAW SCHOOL - Human Rights Clinic: COUNTING DRONE STRIKE DEATHS

7/31/2019 COLUMBIA LAW SCHOOL - Human Rights Clinic: COUNTING DRONE STRIKE DEATHS

1/36

CountingDroneStrikeDeaths

October2012


2/36

2 COLUMBIAHUMANRIGHTSCLINIC

Acknowledgements

ThisreportwasproducedbystudentsandstaffattheHumanRightsClinicatColumbiaLawSchool.TheprimaryauthorandresearcherwasChantalGrut,L.L.M.12,withcontributionsbyNaureenShah,Acting

DirectoroftheHumanRightsClinicandAssociateDirectoroftheCounterterrorismandHumanRights

Project,HumanRightsInstituteatColumbiaLawSchool.Additionalresearchandeditingwereprovided

byLauraONeill,J.D.13,SarahMechlovitzSaadoun,J.D.14,ZoeHutchinson,L.L.M.12andDaniel

Mul,J.D.13.GretaMoseson,programcoordinatorattheHumanRightsInstituteatColumbiaLaw

School,andAshleyKinseth,J.D.13,providedmediaoutreachsupport.

TheColumbiaLawSchoolHumanRightsClinicandtheColumbiaLawSchoolHumanRightsInstituteare

gratefultotheOpenSocietyFoundationsandBullittFoundationfortheirfinancialsupportofthe

InstitutesCounterterrorismandHumanRightsProject,andtoColumbiaLawSchoolforitsongoing

support.

Copyright2012

HumanRightsClinicatColumbiaLawSchool

Allrightsreserved

----------------------------------------------------------

PrintedintheUnitedStatesofAmerica.

Copiesofthisreportareavailablefordownloadat:

http://web.law.columbia.edu/human-rights-institute

HumanRightsClinic,ColumbiaLawSchool

JeromeGreeneHall

435W.116th

St.

NewYork,NewYork10027-7297


3/36

COLUMBIAHUMANRIGHTSCLINIC 3

CountingDroneStrikeDeaths

Summary&Recommendations...................................................................................................................4

Recommendations.......................................................................................................................................7

I.Introduction:DroneDeathEstimatesasFalseAssuranceinPublicDebate.............................................9

II.OurMethodology...................................................................................................................................11

III.WeaknessesofMediaReportsasaMetricforDroneStrikeDeaths....................................................14

IV.OurRecountandEstimateofDroneStrikeDeaths..............................................................................20

V.ComparisonoftheTrackingOrganizationEstimates.............................................................................27

VI.U.S.GovernmentsResponsibilitytoAccountforDroneStrikeDeaths...............................................34


4/36


Summary&RecommendationsPopulardebateonU.S.dronestrikesoftencentersonhowmanyindividualsarekilled,andwhichoftwo

categoriestheindividualskilledfallintomilitantorcivilian.U.S.officialsemphasizetheprecisionof

dronetechnologyandcontendthatextremelyfewcivilianshavebeenkilled.Yetothershavequestioned

theseclaimsandstatedthatthereisevidencetosuggestthatdeaths,andciviliandeathsinparticular,

aremuchhigherthanU.S.officialsadmit.

TheuncertaintyaboutciviliandeathsislargelyduetotheU.S.governmentsresistancetoopenly

providinginformationaboutstrikes.Intheabsenceofofficialdata,themostcommonsourcefordrone

strikecasualtyfiguresisnewsreportsaboutparticularstrikes.Someorganizationshavecataloguedand

aggregatedthesenewsreportstoprovideoverallestimatesofthetotalnumberofindividualskilled,

includingthenumberofmilitantsversuscivilians.TheBureauofInvestigativeJournalism,LongWar

JournalandNewAmericaFoundation(trackingorganizations)areamongthemostinfluentialofsuch

organizations,andtheirworkhasinmanyinstancescatalyzeddebateabouttheeffectivenessand

humanitariancostofstrikes.

Weareconcernedaboutoverrelianceonthetrackingorganizationsestimatesofdronestrikecasualties,

althoughwefindtheestimatesvaluableandagoodfaitheffort.Theestimatesreflectanechochamber

ofsorts:thetrackingorganizationscollectnewsreportsofparticularstrikesandmakeanestimateof

whoiskilledbasedonthem;theseestimatesarethenregularlycitedandrepeatedinsubsequentnews

storiesandmediaanalysispieces.

Inthelimitedpublicdebateondrones,thetrackingorganizationsestimatessubstituteforhardfacts

andinformationthatoughttobeprovidedbytheU.S.government.Wethepublic,theanalystsand

experts,andthepolicymakersstilldonotknowthetrueimpactorhumanitariancostofdrones;the

estimates,thoughwell-intended,mayprovidefalseassurancethatweknowthecostsandcanfairlyassesswhethertocontinuedronestrikes.Furthermore,wherethetrackingorganizationsestimates

significantlyundercountthenumberofcivilianskilledbydronestrikes,theymaydistortourperceptions

andprovidefalsejustificationtopolicymakerswhowanttoexpanddronestrikestonewlocations,and

againstnewgroups.

Becausethestakesarehigh,theColumbiaHumanRightsClinicsetouttothoroughlyexaminethedata

andmethodologyofthethreetrackingorganizations.Ourfindingsaretwo-fold.First,despitethestrong

effortsofthetrackingorganizations,theirestimatesofciviliancasualtiesarehampered

methodologicallyandpractically.Twooftheorganizations,accordingtoourindependentreviewofthe

mediasourcesavailable,significantlyandconsistentlyunderestimatedthepotentialnumberofcivilians

killedinPakistanduringtheyear2011.Second,whilesomeoftheflawsweidentifycanbefixed,others

areinherenttotheprocessandtheseinherentflawsunderscorethattheU.S.governmenthasthe

responsibilitytostepinanddescribeitsownaccountingontheciviliancasualtyquestion.Thetracking

organizationsprovideimportantinformation,butinlightofthemethodologicalandpracticallimitations

weidentify,theirestimatesareaninadequateanddangeroussubstituteforofficialgovernment

estimatesandinformationregardingciviliandeaths.


5/36


Wenotethatsomeofourconclusionsdescribedbelowarecorroboratedbyotherstudies,inparticular,

aSeptember2012reportbythehumanrightsclinicsatNYUSchoolofLawandStanfordLawSchoolthat

examinedadistinctandmorerecentdataset.Takentogether,thevariousstudiesunderscoretheneed

foraU.S.governmentaccountingofdronestrikedeaths,aswellasgreatercarebymediaoutletsand

observersindescribingtheimpactofdronestrikes.

FlawsinMediaReportingonDroneStrikeDeaths:Weanalyzedthetrackingorganizationsdata

collectionforstrikesinPakistanduring2011andfoundthatwhiletheirestimatesareuseful,they

necessarilyreflectthebiasesandflawsoftheirmediareportsources,i.e.,thenewsstoriesabout

particularstrikeswhichtheyaggregatetoarriveattheirownestimates.Mediacoverageofdronestrikes

isinconsistent,anditislikelythatsomedeathsandevenentirestrikesarenotcapturedbytracking

organizations,particularlytotheextenttheyrelyonEnglish-languagemediasources.Themediareports

oftenrelyonverylimitedsources,inparticularthewordofanonymousPakistaniofficialswhowillnot

puttheirnamestoastatement.Thereisnostandarddefinitionthatthemediasourcesusetocategorize

apersonasmilitantoracivilian,norastandardizedmeasurebywhichthemediasourcesweighand

corroboratetheirinformation.

OurRecount:Examiningthesamemediareportsthatthetrackingorganizationscited,wefounda

significantlyhighernumberofreporteddeathsoverallandciviliandeathsinparticularthanisreflected

inthecountsoftheNewAmericaFoundationandLongWarJournal;ourcountwassimilartothatofthe

BureauofInvestigativeJournalism.Ofthethreeorganizations,onlytheBureauhasconsistently

purportedtoactivelytrack civiliancasualtiesasopposedtofocusingonprovidinganestimateofthe

overallnumberofindividualskilled.Nevertheless,newsanalysisandpoliticalcommentaryfrequently

citeNewAmericaFoundationandLongWarJournalsnumbersinmakingconclusionsabouttheimpact

ofdronestrikesonciviliansandlocalcommunities.Exclusiveorheavyrelianceonthecasualtycountsof

thesetwoorganizationsisnotappropriatebecauseofthesignificantmethodologicalflawsweidentify.WhilewedonotagreewiththeBureausanalysisofmediasourcesinallcases,itappearstohavea

moremethodologicallysoundcountofciviliancasualties,commensuratewithitsspecialfocusonthat

issue.

Ourrecountfoundreportsofbetween72and155civilianskilledin2011Pakistandronestrikes,with52

ofthereportedlyciviliandeadidentifiedbynamearelativelystrongindicatorofreliability .By

comparison,NewAmericaFoundationscountisjust3to9civilianskilledduringthisperiod;LongWar

Journalscountis30civilianskilled.Inpercentageterms,andbasedontheirandourminimumnumbers,

wecounted2300percentmoreciviliancasualtiesthantheNewAmericaFoundation,and140percent

moreciviliancasualtiesthanNewAmericascivilianandunknowncasualtycountscombined.We

counted140percentmoreminimumciviliancasualtiesthantheLongWarJournal.TheBureauof

InvestigativeJournalismscountofbetween68to157civilianskilledinPakistanduring2011isclosestto

ourown:wecountedonly5.9percentmoreminimumciviliancasualties.Wedescribeourstandardsfor

therecountbelow.

Itisimportanttonotethatdespitethegreatcarewetookinreviewingthedataandoriginalmedia

sources,ourrecountdoesnotpurporttobereliableoranaccurateindicatoroftheactualnumberof


6/36


civilianormilitantcasualtiesofU.S.dronestrikes.Rather,ourrecountreflectstheextenttowhich

civiliancasualtiesthatarecrediblyreportedarenonethelessmissedbythetrackingorganizations;it

underscoresthatthepublicandpolicymakersdonothavecompleteinformationaboutthe

humanitariancostofdronestrikes.

DiscrepanciesinTrackingOrganizationsEstimates:Incomparingthetrackingorganizationsstudies,wefoundthattheirmethodologiesvary,leadingeachofthemtoarriveatdifferentestimatesofthenumber

andidentityofindividualskilled.Weidentifiedfourprincipalreasonsforthediscrepanciesintracking

organizationsoverallcasualtycountsandcountsperstrike:(1)thenumberofmediasourcesthe

trackingorganizationsreliedon;(2)theBureauofInvestigativeJournalismsuseofnon-mediasources;

(3)theNewAmericaFoundationandLongWarJournalsdecisionnottoupdatefiguresinsomecases

wheremediasourceschangetheircasualtycountsinsubsequentreports;and(4)thetracking

organizationsdecisionsnottoincludesomedatafromsomeoftheirownlistedsources.

U.S.GovernmentsResponsibilitytoProvideEstimates:Thediscrepanciesincountsbythetracking

organizationscredibleandwell-resourcedinstitutionsunderscorethedifficultyofgainingan

accurateunderstandingoftheimpactofdronestrikesfrommediareportsalone.Thepublicandsome

policymakersarecompelledtorelyontheseestimatestojudgetheimpactofdronestrikesbecausethe

U.S.governmenthasnotofficiallyprovidedinformationondronestrikedeaths.Whiletoutingthe

successofthedroneprogramandparticularhigh-profilestrikes,U.S.officialshaveavoidedproviding

specificsandcitednationalsecurity.ThepublichasnoinformationonhowandwhethertheU.S.

tracksandinvestigatespotentialciviliandeaths.

Yet,inotherconflictsettingssuchasAfghanistan,U.S.officialshaveprovidedsomeofthisinformation

withoutcompromisingU.S.security.AstheU.S.governmentanticipatesthecontinuedandexpanded

useoflethaldronetechnology,itowesthepublicagenuineassessmentoftheimpactofdronestrikes,

includingtheeffectsonlocalcivilianpopulations.TheU.S.governmentshould,totheextentpracticable,

provideitsownestimateofthenumberofindividualskilledandtheiridentities.Moreover,insteadof

seekingtodiscrediton-the-groundreportingbyjournalistsandhumanrightsgroupsthatputsforward

evidenceofciviliancasualties,theU.S.governmentshouldinvestigateandaddressthereportedstrikes

andcasualties.

Media&TrackingOrganizationsRoles:Weencouragemediaandobserverstoreconsiderthewaythey

addressthequestionofwhoisbeingkilledindronestrikes.Theyshouldqualifytheiruseofestimates

providedbythetrackingorganizations.Theseestimatesarenotactualbodycountstheyarereports

fromaregionwhereevenseasonedjournalistsandinvestigatorssufferfromlimitedaccess,andwhere

witnessstatementsandofficialsexplanationsmaybebiased.Whenmediaandcommentatorsuse

civiliancasualtyestimates,theyshouldacknowledgetheselimitationsandtheambiguityoftermslike

militant.Likewise,trackingorganizationsshouldacknowledgethelimitationsoftheirstudies,and

addressthemethodologicalproblemswehaveidentified.


7/36


Recommendations

TOTHEU.S.GOVERNMENT

1.

Totheextentpracticable,collectandreleaseestimatesregardingpastdronestrikecasualtiesandprovideinformationonnewstrikesafteroperationsarecompleted,includingthefollowing

information:

a. ThetotalnumberofindividualskilledorinjuredinU.S.dronestrikesoutsideofAfghanistan

b. ThetotalnumberofindividualskilledwhomtheU.S.identifiesascombatantsorpersonssubjecttodirectattackunderU.S.legalstandardsorU.S.interpretationsof

humanitarianlaw.

c. ThetotalnumberofindividualskilledwhomtheU.S.identifiesasnotsubjecttodirectattackunderU.S.legalstandards,including:civilianskilledcollaterally(i.e.killedinan

attackonamilitaryobjective,andnotidentifiedasaciviliansdirectlyparticipatinginhostilities);civilianskilledmistakenly(e.g.killedbasedonmistakenidentity);or

individualsotherwiseaffordedprotectionfromdirectattackunderU.S.legalstandards

orU.S.interpretationsofhumanitarianlaw.

d. Thetotalnumberofdronestrikesandthelocationanddateofeachdronestrike.2. DisclosethelegalstandardsanddefinitionstheU.S.governmentusesincategorizingthe

individualskilledasmilitarytargetsorindividualssubjecttodirectattack;civilianssubjectto

directattack(e.g.civiliansdirectlyparticipatinginhostilities);civiliansentitledtoprotectionand

killedcollaterally.

3. Investigatecrediblereportsofciviliandeathbymediaandnongovernmentalorganizations,andmakeamendstofamiliesandlocalcommunitieswhereappropriate.

TOTRACKINGORGANIZATIONS

1. Publishestimateswithaprominentdisclaimer,includinginregardtothesourcesforitsdataset,thevarianceofreportingamongnewsoutletsandovertimeandthepotentialbiasesofprimary

sourcesonwhichidentificationofmilitantsversusciviliansarebased.

2. Fororganizationsthatrelyonalimitedsetofmediasources,expandtherangeofmediasourcesincataloguingdeaths;wheremoremediareportsareunavailable,qualifyestimatesas

incompleteoruncorroborated.

3. IndescribingthefindingsofmediareportsthatciteunnamedPakistaniofficials,acknowledgethepotentialpoliticalbiasorusequalifyinglanguagesuchasallegedandattributivelanguage

suchasaccordingto.Whereadecisionismadetodiscardcertainreportedfiguresinany

particularstrikeestimate,indicatewhichreportwasdiscardedandexplainwhy.


8/36


TOMEDIAANDCOMMENTATORS

1. Whendescribingtheoverallnumberandidentityofindividualskilledbydronestrikes:a. ReportedDeaths:Acknowledgethatinformationaboutdronestrikecasualtiesis

limitedasageneralmatter,anddescribethetrackingorganizationsestimatesas

collationsofreporteddeathsinacontextwherevirtuallynomediareportsarebasedoninformationgatheredinsidetheregionfirsthandorabletobeverifiedbythemedia

organizationitself,andwheremediareportssometimesrelyonbiasedsources,e.g.,

anonymousgovernmentofficials.

b. DifferentEstimates:Whencitingasingletrackingorganizationsestimateofthenumberoridentityofindividualskilled,acknowledgewheretherearediscrepanciesbetween

thatestimateandestimatesbyotherorganizations.

2. Whenreportingonparticularstrikes:a. Limitedreporting:Whereappropriate,acknowledgelimitsofreportingandinformation

aboutastrike,e.g.,thatinformationprovidedbylocalsourcescouldnotbeverifieddue

tolimitedaccesstotheregion.b. MilitantDeaths:

i. Regardingestimatesofthenumberofmilitantsversuscivilianskilled,acknowledgethatthedeterminationofwhetheracasualtyiscategorizedas

militantorcivilianisambiguousandcontroversial,e.g.,byusingtheterm

alleged.

ii. Inrecognitionofitsambiguousandcontroversialcharacter,avoidusingthewordmilitantunlessquotingagovernmentofficial;usemorespecific

identifierswherepossible.


9/36


I.Introduction:DroneDeathEstimatesasFalseAssurancein

PublicDebateDebateaboutlethaldroneoperationsbytheUnitedStatesoftencentersonthenumberofresulting

casualties,andwhichoftwocategoriesindividualskilledfallintomilitantorcivilian.Inexplainingand

justifyingtheexpansionofU.S.dronestrikesbeyondAfghanistan,U.S.officialsemphasizetheprecisioncapabilitiesofdronetechnologyandlimitednumberofciviliancasualties.

1Yetothershavequestioned

theseclaimsandstatedthatthereisevidencetosuggestthatdeaths,andciviliandeathsinparticular,

aremuchhigherthanU.S.officialsadmit.2

AlthoughtheObamaadministrationhasrecentlyacknowledgeditscovertdronestrikecampaignsin

PakistanandYemen,itcontinuestoavoiddisclosureofbasicdetailsincludingthenumberandidentity

ofindividualskilled.BecauseofU.S.governmentsecrecy,themostcommonsourceforcasualtyfigures

ismediareportsfrominternationalandlocalnewsoutlets.Mediareportsofparticularstrikesare

publishedbyarangeofoutlets:prominentWesternnewspapers,internationalwireservices,regional

newsagenciesandPakistaninewspapers.NewAmericaFoundation,LongWarJournalandBureauofInvestigativeJournalism(trackingorganizations)areamonganumberofotherorganizationsthattrack

thesemediareportsand,catalogingthereports,provideestimatesofthetotalnumberofindividuals

killed,includingthenumberofmilitantskilled.

Overthelastyearespecially,thetrackingorganizationsestimateshavepermeatedandsignificantly

impacteddebateondronestrikes.Newsanalysisandcommentarybypoliticalobserversandexperts

oftencitetheseestimates.Evenhumanrightsexperts,suchastheU.N.SpecialRapporteuron

1PresidentObamahasdescribeddronestrikesasprecise,precisionstrikesagainstAlQaedaandtheir

affiliates.SeeChristiParsonsandMichaelA.Memoli,ObamaOpensUpaboutDroneStrikesinPakistan,L.A.Times,January31,2012(quotingObamainavirtualinterviewonGoogle+and

YouTube).Obamaadministrationofficialshaverepeatedlyemphasizedthatdroneandotheradvanced

technologycanensurethatthebestintelligenceisavailableforplanningandcarryingoutoperations

withtheresultthattheriskofciviliancasualtiescanbeminimizedoravoidedaltogether.Various

administrationofficialshavespokeninnearlyverbatimterms.SeeAttorneyGeneralEricHolder,

AttorneyGeneralEricHolderSpeaksatNorthwesternUniversitySchoolofLaw(Mar.5,2012)available

athttp://www.justice.gov/iso/opa/ag/speeches/2012/ag-speech-1203051.html/ (Infact,theuseof

advancedweaponsmayhelptoensurethatthebestintelligenceisavailableforplanningandcarrying

outoperations,andthattheriskofciviliancasualtiescanbeminimizedoravoidedaltogether.);Harold

Koh,LegalAdviser,U.S.DepartmentofState,SpeechattheAnnualMeetingoftheAmericanSocietyof

InternationalLaw,(Mar.30,2010)availableathttp://www.state.gov/s/l/releases/remarks/139119.htm(Indeed,usingsuchadvancedtechnologiescanensureboththatthebestintelligenceisavailablefor

planningoperations,andthatciviliancasualtiesareminimizedincarryingoutsuchoperations.);Jeh

Johnson,NationalSecurityLaw,LawyersandLawyeringintheObamaAdministration,RemarksatYale

LawSchool(Feb.22,2012),availableatwww.lawfareblog.com/2012/02/jeh-johnson-speech-at-yale-

law-school/(Advancedtechnologycanensureboththatthebestintelligenceisavailableforplanning

operations,andthatciviliancasualtiesareminimizedincarryingoutsuchoperations.).

2See,e.g,ScottShane,C.I.A.IsDisputedonCivilianTollinDroneStrikes,N.Y.Times(Aug.11,2011).


10/36

1

COLUMBIAHUMANRIGHTSCLINIC

extrajudicial,summary,orarbitraryexecution,refertotheestimates.3Scholarshipandthinktank

studiestaketrackingorganizationsestimatesasastartingpointforfurtheranalysis.4

OntheirwebsitesandininterviewsandexchangeswithColumbia,noneofthetrackingorganizations

researcherspresenttheirestimatesasdefinitive.Webelievethetrackingorganizationsareengagedina

goodfaithefforttoinformandencouragepublicdebate,andhavenotsetouttoprovideconclusiveanswers.Nevertheless,intheabsenceofotherinformation,theseestimatestransmuteintosomething

likeactualbodycounts.Amongconcernedpolicymakersandobservers,theestimatesaresometimes

discussedasthoughtheywerebasedoncredible,corroboratedandfirsthandresearchratherthana

compilationofnewsarticlesandothermaterialthatonlyrarelyincludeson-the-groundresearch.

Theproblemisthatastheestimatesareassimilatedintofact,theythreatentobecomewhateverybody

knowsabouttheU.S.covertdronestrikesprogram .Theestimatesprovideadangerousassurance:the

humantollissomethingwehaveidentified,debatedandconsidered.Ifweknowwhoandhowmany

peoplewehavekilled,callstoexamineanddeliberateonthedroneprogramandcallstoenditlose

theirurgency.Wemaycometofalselybelievethatcovertdronestrikesareanopensecretwhenin

fact,theU.S.governmentcontinuestoresistdisclosureofbasicandimportantinformationaboutthe

dronestrikesprogram.Moreover,wherethetrackingorganizationsestimatessignificantlyundercount

thenumberofcivilianskilledbydronestrikes,theymaydistortourperceptionsandprovidefalse

justificationtopolicymakerswhowanttoexpanddronestrikestonewlocations,andagainstnew

groups.5

OurstudyunderscorestheneedforresponsibleengagementbytheU.S.governmentontheissueof

dronestrikes,andinparticular,theimpactonlocalcivilianpopulations.Itbuildsontheworkofscholars

andobserverswho,basedontheirfamiliaritywithreportingprocessesandlocaldynamics,haveforthe

pastfewyearsquestionedthereliabilityofdronestrikedeathestimates.6Whiletheestimatesof

3ChristofHeyns,SpecialRapporteuronextrajudicial,summaryorarbitraryexecutions,Follow-upto

countryrecommendationsUnitedStatesofAmerica,30March2012,A/HRC/20/22/Add.3,atpara.

81.4See,e.g.,PatrickB.Johnson&AnoopSarbahi,"TheImpactofU.S.DroneStrikesonTerrorismin

Pakistan,"(February25,2012),availableat http://patrickjohnston.info/materials/drones.pdf5

Foradiscussionofdronestrikesandtheexpansionofgroupsandindividualsofwhomaybetargeted,seeColumbiaHumanRightsClinic&CenterforCiviliansinConflict, TheCivilianImpactofDrones:

UnexaminedCosts,UnansweredQuestions67-77(2012).6Thisreliancehassparkedcritiquesbyjournalistsandscholarsskepticalofthereliabilityofthetracking

organizationscountsandmediareports.See,e.g.,ConorFriedersdorf,FlawedAnalysisofDroneStrike

DataisMisleadingAmericans,TheAtlantic(Blog)(July18,2012);ChaseMadar,SearchforaMethod,

TheNewInquiry.com(July2,2012);MuhammadIdreesAhmad,TheMagicalRealismofBodyCounts,

Aljazeera.com(June13,2011);NYUCenterforHumanRightsandGlobalJustice&StanfordInternational

HumanRightsClinic,LivingUnderDrones43-54(2012).


11/36


Pakistan-basedtrackingorganizationsareoutsideofthescopeofthisreport,theiruseofawiderrange

ofsources,suchaslocalmediareportsinlocallanguagesandhospitalrecords,isinstructive.7

II.OurMethodology

ThisreportbuildsonpreviousstudiesbytheColumbiaHumanRightsClinicandHumanRightsInstitute

regardingtheimpactofdronestrikesonlocalcommunitiesandtheambiguityinU.S.legalstandards

regardingwhomaybetargetedlethally.8

Inthisreport,wesetouttoexaminetheassumptionsandaccuracyofinformationaboutdronestrike

deaths,includingtheidentityoftheindividualskilled.NewAmericaFoundation,LongWarJournaland

BureauofInvestigativeJournalismeacharriveatdifferentestimatesofthetotalnumbersofindividuals

killedandtheiridentities.Tofindoutwhy,wereviewedeachorganizationsdataforallstrikesin

Pakistanin2011,includingthemediareportstheyreferenced,andindependentlycountedreported

casualties.OurstudyconsistedofallreporteddronestrikesinPakistanin2011asignificantand

discretedatasetthatallowsustocompareourresultstothoseoftheotherorganizations,strike-by-strike,andinend-of-yearstatistics.Ourcompletestrike-by-strikecomparison,analysisanddatasetis

availableonline.

Inadditiontotheanalysisandcritiqueofmethodsinthisreport,werecountedthereportedcasualties

foreachstrikeusingeachtrackingorganizationsownsourcesandarrivedatourownestimates.We

usedonlythesourceslistedbytheorganizations,whicharehyperlinkedontheirwebsites(orinthecase

ofNewAmerica,providedinaseparatePDFdocumentwhichishyperlinkedontheirwebsite).Wedid

notseekoutadditionalsources,butwhereahyperlinkwasbrokenandappearedtobethesourceofan

organizationsupperorlowercasualtyfigure,wehavetriedtore-sourcethearticle.

Wegaveeachorganizationanopportunitytoreviewandrespondtoourdataandfindings.BillRoggioof

theLongWarJournalspokewithusextensivelyabouttheorganizationsmethodology;theBureauof

7Wedonotfocusonthesestudiesinpartbecauseinternationalmediararelycitetheirfigures,butalso

becausetheircountsarebasedonUrduandotherlanguagesourcesthatwedonothavetheresources

toverify.Thethreemostwell-knownPakistantrackingorganizationsarePakistanInstituteforPeace

Studies,PakistanBodyCountandConflictMonitoringCenter.Accordingtoitswebsite,thePakistan

InstituteforPeaceStudiesConflictandSecurityDatabaseteammonitorsaround100localandnational

newspapers,magazinesandjournalsonadailybasis,relyingheavilyonregionalcorrespondentsto

cross-checkinformationandseekfurtherdetailsofissuescoveredbythemedia.SeePakInstitutefor

PeaceStudies,DigitalDatabaseonConflictandSecurity,availableathttp://www.san-pips.com/app/database/index.php?id=3(subscriptiononly);EveryCasualty,InternationalPractitioners

Network,PakInstituteforPeaceStudies,availableathttp://www.everycasualty.org/practitioners/pips

seealsohttp://san-pips.com/index.php?action=reports&id=tml3 ;PakistanBodyCount,availableat

http://pakistanbodycount.org/drone_attack;ConflictMonitoringCenter,availableat

http://cmcpk.wordpress.com/drone-attacks-in-pakistan/.8SeeColumbiaHumanRightsClinic&CenterforCiviliansinConflict, TheCivilianImpactofDrones:

UnexaminedCosts,UnansweredQuestions(2012);HumanRightsInstitute,ColumbiaLawSchool,

TargetingOperationswithDroneTechnology:HumanitarianLawImplications(March25,2011).


12/36

1


InvestigativeJournalismsChrisWoodsreviewedandcommentedonourdatafindingsandspokewith

usextensivelyabouttheBureausmethodology.Wearegratefulfortheirinsightsandcomments.

DespitenumerousattemptstodiscussourfindingswithNewAmericaFoundation,theydeclinedto

comment,butdidsubstantiallyaltertheirdataafterweprovidedourdetailedstrikecounts.9

TheonlyorganizationthatchosetoresponddirectlytousabouteachstrikecountwherewereachedadifferentresultwastheBureauofInvestigativeJournalism.Someinitialdifferencesbetweenourcounts

havebeeneliminatedthroughthisprocess,astheBureauchangeditsdatabasedonsomeofourwork,

andviceversa.Incaseswherewecontinuetodisagree,wehaveincorporatedtheBureausspecific

responsesinourdataset.

Toarriveatourindependentrecount,wetookintoaccountthelowestandhighestreportednumbers

presentedbylistedmediasources,exceptincaseswhereweweresatisfiedthatcertainfigureshad

beensupersededbylaterreportsor,inafewcases,wheretheycontainedevidenttypographical

errors.10

Asdiscussedbelow,webelievethatthisistheappropriateandmostaccurateapproach.We

havetriedtodiscardoutdatednumbersveryconservatively,doingsoonlywhenthereisaclear

differenceovertimeandwherethesamenewsagenciespublishupdatedfigures.Inourview,asthe

majorityofmediareportsprovidesolittleinformationandrelyonsofewsources,itisararecasewhere

areportedfigurecanbeconfidentlydiscardedonanyotherbasis.Wehavetriedtobeastransparentas

possibleabouthowwehavereachedourcountanditscomparisontoallofthereportedfigureswithin

thestrike-by-strikeanalysis.

Incaseswherewehadconcernswiththemediasources,wehavetakenalowerfigureof0.Incases

wherethereareconflictingreportsabouttheidentityoftheindividualskilledaseithermilitantsor

civilians,ourfiguresreflectbothpossibilitiesforexample,if5peoplewerereportedkilled,wetakea

lowerfigureof0andanupperfigureof5inbothcategories.11

Itisimportanttonotethatdespitethegreatcarewetookinreviewingthedataandoriginalmedia

sources,ourestimatesdonotpurporttobereliableoraccurateindicatorsoftheactualnumberof

civilianormilitantcasualtiesofU.S.dronestrikes.TheU.S.governmentshouldprovidesuchfigures

9InanemailresponseonJuly30,2012,PeterBergenoftheNewAmericaFoundationdeclinedtospeak

withus,stating:Wepublishallofourdataandupdateitcontinuouslyonourwebsiteandsoany

questionsyoumighthaveshouldbeansweredbywhatisonthesite.Followingseveralrepeated

inquiries,inanemailonAugust12,2012,JenniferRowlandoftheNewAmericaFoundationaskedfor

ourwrittenfindingsandonAugust15,2012indicatedtheywereunderreview.Despiterepeated

attemptsatcontactbyemailandphone,NewAmericahasnotrespondedwithcommentsorotherinformation.However,inlateAugust2012,NewAmericasignificantlychangedthewaythatitsdatasets

arerecordedandchangedanumberofindividualstrikeaccounts,perthespecificdatadiscrepanciesour

findingshadidentified.10

Ourrecountsupperandlowerfiguresarebasedonlyonmediasources,allofwhicharelistedas

footnotesinourdataset.Themediasourcesdonotincludeunknownblogsorwebsitespurportedlyrun

bymilitantgroups.11

Thisisalsothereasonwhyourminimumcivilianandmilitantcasualtyfiguresdonotadduptoour

minimumfigurefortotalkilled.


13/36


andclarifyitsstandardsforthepermissibilityofdirecttargeting.Rather,ourrecountreflectstheextent

towhichciviliancasualtieswhicharecrediblyreportedarenonethelessmissedbythetracking

organizations;itunderscoresthatthepublicandpolicymakersdonothavecompleteinformationabout

thehumanitariancostofdronestrikes.


14/36

1


III.WeaknessesofMediaReportsasaMetricforDroneStrike

DeathsMediareportsareatpresentthebestandperhapsonlywaytogetapictureofthefrequencyandscope

ofU.S.dronestrikesoutsideofAfghanistanandanideaoftheoverallnumberofpeoplebeingkilledby

them.However,therearesystematicflawsandbiasesinthereporting.Whileinsomerarecasesthereis

in-depthreportingonthecircumstancesofparticularstrikes,thisisunfortunatelyfarfromthenorm.

LimitedMediaAccessandFewDetails

Mediareportsofparticulardronestrikesarebasedonlimitedreporting,withthesamefewjournalists

andnewsoutletsprovidingthesamematerialstomultiplewireagenciesandnationalorinternational

press.Moreover,mediareportsareusuallybasedonlimitedon-the-groundinvestigation(withthe

exceptionofsomePakistaninewspapers,whichwenotebelow).Widerandmorein-depthreportingis

typicallyreservedforcaseswhereahigh-levelmilitantleaderhasbeenreportedkilled,orcaseswithan

unusuallyhighnumberofoverallandreportedlyciviliancasualties.Reportsoftenprovidenomore

informationthanthelocationofthestrike,theallegedorapparenttarget(suchasacompoundora

vehicle),thenumberofpeoplereporteddeadandanofficialclaimthatthosekilledweremilitants.

Whilethepublicandpolicymakersmayviewinternationalmediareportsasespeciallycredible,these

outletsincludingtheNewYorkTimes,theBBCandwireservicessuchasReutersandtheAssociated

Pressaregenerallyunabletoaccessthetribalareaswheredronestrikesareoccurring,makingthem

reliantonlocaljournalistsorstringersfortheirreporting.Internationalmediasometimesreportthe

numberandidentityofthosekilledbasedonthereportsofseverallocalstringers.However,the

stringersthemselvesmaybeunabletogototheareawhereadronestrikeactuallyoccurredduetosecurityissues,makingtheirreportssubstantiallydependentonthewordofahandfuloflocalofficials

aproblemweexplorefurtherbelow.Evenwhenstringersareabletoreachtheareaswheredrone

strikesoccurredandconducttheirownon-the-groundreporting,resultingreportsmayfailtoreflect

possibleciviliancasualties;ifonlyonestringerfindsevidenceofciviliancasualties,comparedtothreeor

fourotherstringerswhodonot,internationalmediawillordinarilynotreportthepossibilityofcivilian

casualties.12

Ontheotherhand,reportsofciviliancasualtiesbasedonstatementsofunnamedwitnessesorlocal

villagersmaybeflawed.Soonafterastrike,localmilitantgroupsmaymovethebodiesofindividuals

killed,forburialortoobscuretheidentityofthosekilled.Witnessesmayfearretaliationbylocalmilitant

groupsiftheydiscusswhatoccurred;itispossiblethatlocalgovernmentofficials,covertagents,or

12

SeeAsiaPacificForumWBAI99.5FM,KillerDrones:CountingtheHumanCosts(Interviewwith

journalistMadihaTahir),audioavailableathttp://www.asiapacificforum.organdtranscriptonfilewith

ColumbiaHumanRightsClinic;seealsoNYUCenterforHumanRightsandGlobalJustice&Stanford

InternationalHumanRightsClinic,LivingUnderDrones37-38(2012).


15/36


militantgroupscouldinfluencewitnesseseitheragainstprovidinginformationatall,ortoward

providingmisinformationaboutwhoiskilledalthoughwedonotknowofreportstothiseffect.

WhoDefinesMilitantsversusCivilians?

Reportingondronestrikecasualties,andthetrackingofthosereports,typicallydividethedeadintwo

categories:militantsandcivilians.Thehiddenbiasofthiscategorizationstemsfromtheambiguity

oftheterms:theyarenotdefinedbytheU.S.government,thoughU.S.officialsusethem;theterms

soundvaguelylegal,althoughtheyonlylooselytracklegalandscholarlydebatesaboutwhomaybe

lawfullytargeted.Withoutauniversallyacceptedorstandardizeddefinitionfortheseterms,

categorizationofmilitantandciviliandeathsisbiasedbythedefinitionoftheindividualstowhom

mediareportsciteforidentifyingthedead.WhethertheseprimarysourcesareunnamedU.S.and

Pakistaniofficials,orunnamedlocalvillagersandwitnesses,identificationofthosekilledasmilitants

orciviliansislikelydrivenbypoliticalinterests,andcoloredbytheperspectiveandexperiencesofthe

source.Asthesetermsappearinmediareportsandthetrackingorganizationstudies,theymightbebetterunderstoodasmoralcategoriesofwhoshouldandshouldnotbekilled.Theyare,tothatextent,

inherentlylimitedandbiased.13

Ourreviewofthemediareportsandtrackingorganizationstudiessuggeststhattheyregularlyuse

civiliantorefertoapersonforwhomthereisnoallegationofaffiliationwithamilitantgroup,and

militantforapersonforwhomthereis.Ascolloquialexpressions,theseunderstandingsappear

intuitive.Thus,inmediaandtrackingorganizationreports,namedandwell-knownmilitantgroup

leadersarecategorizedasmilitantsandyoungchildrenandwomenareoftencategorizedas

civilians.

Thedistinctionbetweenmilitantandcivilianismoremuddledandcontroversial,however,than

theseexamplesreflect.14

Theslippagebetweenmilitantandcivilianascolloquialtermsandlegal

termscausesconfusionandmaybeareasonfordifferentaccountingofwhoisacivilian.Asalegal

matter,undersomecircumstancesciviliansmaylawfullybetargetedperhapsleadingU.S.and

Pakistanigovernmentsourcestorefertosuchciviliansasmilitants.15

Thecircumstancesunderwhich

13

Otherobservershavealsomadethisconclusion.See,e.g.,NYUCenterforHumanRightsandGlobal

Justice&StanfordInternationalHumanRightsClinic, LivingUnderDrones30-31(2012).14

Eventheexamplesofmilitantleadersandyoungchildrenandwomencarrycontroversy.Some

commentatorswouldarguethatmilitantleaderscannotbetargetedwhiletheyarenotdirectly

participatinginhostilities.Moreover,civilianwomenandchildrencouldbetargetableunderhumanitarianlaw,althoughyoungerchildrencouldpresumablynotformtheaffiliationsorintentions

necessarytoloseprotectionascivilians.SeeAvrilMcDonald,TheChallengestoInternational

HumanitarianLawandthePrinciplesofDistinctionandProtectionfromtheIncreasedParticipationof

CiviliansinHostilities,AsserInstitute(April2004),availableat

http://www.asser.nl/default.aspx?site_id=9&level1=13337&level2=13379 ;infranote15.15

Theinternationalhumanitarianlawprincipleofdistinctionrequirespartiestoaconflicttodistinguish

inattackbetweencombatants,asdefinedinArticle4oftheThirdGenevaConvention,andcivilians.In

aninternationalarmedconflict,individualswhoarenotmembersofthearmedforcesareciviliansand


16/36

1


civiliansloseprotectionunderhumanitarianlawandbecomesubjecttodirectattackisamatterofhotly

contesteddebateamonglawyersandscholarsintheU.S.andinternationally.16

Governmentsources

mightprovidelowerciviliancasualtyestimatesthanlocalwitnessesbecauseinternalgovernment

standardspermittargetingofanexpansiverangeofcivilianssuchascivilianswho,thoughnot

membersofamilitantgroup,aresuspectedofsomeaffiliationorofprovidingsomematerialsupportto

militantgroupssometimesbeyondconventionalinterpretationsofhumanitarianlaw.

Yetthereareassumptionsandbiasesatplayindefiningsuchindividualsasmilitantsorinsistingthey

arecivilians:ontheonehand,governmentsmayjustifytargetingbasedonanindividualsprovisionof

suppliestoalocalmilitantgroup,whileontheotherhandlocalwitnessesmightcharacterizethe

individualasinnocentandnotethatthesupplieswerefoodormedicine,orprovidedonlyunder

duress.17

Likewise,governmentsmayjustifytargetinglocalgroupswhoaremeetingormixingwith

individualsidentifiedasmilitants;yetlocalwitnessesandobserversmightcharacterizethosemeetings

andtalksasattemptsatreconciliationorpeace-building,anddecrydronestrikesfortargetingcivilians.

areentitledtoprotectionagainstdirectattack.Inanon-internationalarmedconflict,acustomaryrule

ofdistinctionapplies,whichisformulatedinsimilarterms.SeeGenevaConventionRelativetothe

TreatmentofPrisonersofWar,Art.4,Aug.12,1949,6U.S.T.3316,75U.N.T.S.135(defining

combatant);ProtocolAdditionaltotheGenevaConventionsof12August1949,andrelatingtothe

ProtectionofVictimsofInternationalArmedConflicts(AdditionalProtocolI),art.50-51,June8,1977,

1125U.N.T.S.3,23[hereinafterAdditionalProtocolI];Rules1and3,ICRC,CustomaryInternational

HumanitarianLawDatabase,http://www.icrc.org/customary-ihl/eng/docs/home [hereinafterICRC,

CustomaryLawDatabase].TheU.S.isnotpartytoAdditionalProtocolI,butregardssomeofits

provisionsascustomarylaw.Inthisreport,werefertocustomarylawasrecognizedbyICRCsstudyon

customarylaw,althoughitsviewsdonotalwaysreflectthoseoftheU.S.government.Thereis

nonethelesssubstantialdebateabouthowtocategorizeindividualswhomaybetargetedbydrone

strikes,asmembersofnon-statearmedgroupsorciviliansdirectlyparticipatinginhostilities.16

Amongtheareasofdebate,twoofthemosthighlycontestedissuesare:(1)whomaybetargetedas

directlyparticipatinginhostilitiesandforhowlong;and(2)whomaybetargetedasfulfillinga

continuouscombatantfunction,astatusbywhichmembersoforganizedarmedgroupsceasetobe

civiliansandloseprotectionagainstdirectattack.Forabriefsummaryofpositionsandcontroversies,

seeHumanRightsInstitute,ColumbiaLawSchool, TargetingOperationswithDroneTechnology:

HumanitarianLawImplications(March25,2011),15-23availableat

http://www.law.columbia.edu/ipimages/Human_Rights_Institute/BackgroundNoteASILColumbia.pdf

(discussingandreferencing,interalia,ICRC,InterpretiveGuidanceontheNotionofDirectParticipation

inHostilitiesUnderInternationalHumanitarianLaw(NilsMelzered.,2009);MichaelSchmitt,

DeconstructingDirectParticipationinHostilities:TheConstitutiveElements,42N.Y.U.J.IntlL.&Pol.

697,699(2010);5RyanGoodman&DerekJinks,TheICRCInterpretiveGuidanceontheNotionofDirectParticipationinHostilitiesunderInternationalHumanitarianLaw:anIntroductiontotheForum,42

N.Y.U.J.IntlL.&Pol.637,640(2010);KennethWatkin,Opportunitylost:organizedarmedgroupsand

theICRCDirectParticipationinHostilitiesInterpretiveGuidance,42N.Y.U.J.IntlL.&Pol.640,692

(2010);NilsMelzer,KeepingtheBalanceBetweenMilitaryNecessityandHumanity:AResponsetoFour

CritiquesontheICRCsInterpretiveGuidanceontheNotionofDirectParticipationinHostilities,N.Y.U.

J.IntlL.&Pol.831,833(2010)).17

SeeColumbiaHumanRightsClinic&CenterforCiviliansinConflict, TheCivilianImpactofDrones:

UnexaminedCosts,UnansweredQuestions41-42(2012).


17/36


Ourreviewofmediareportsfrom2011suggeststhat,beyondthechildrenandnamedmilitantleaders,

thereisrarelyenoughinformationprovidedinmediareportsforanoutsiderincludingatracking

organizationtoindependentlyassesswhethertheuseofthemilitantorcivilianlabelisaccurate,

accordingtoitsownstandardsorpresumedlegalstandards.

ArareexampleofdeepermediareportingontheidentitiesofthosekilledfollowedastrikeonBarmal,

SouthWaziristan,onJuly12,2011,18

butthegreaterlevelofdetailunderscoresourpointthatthelabels

militantandcivilianmayturnontheperspectiveoftheprimarysources,andaredifficultfortracking

organizationstoindependentlyassess.Mediareportscitedanonymousofficialsasindicatingthestrike

killedmilitants.PakistanTodayprovidedfurtherdetail,stating5youngsters,associatedwiththe

HaqqaniNetworkwerekilled,anditnamedthreeofthem.DigitalJournalreportedthatastatementin

anUrdudailyfromoneofthedeceasedsuniversityprofessorsdescribedhimasaninnocent,intelligent

studentwhohadcountlessgoodqualitiesandwhodevotedallhistimetohisstudies.Thestatement

fromtheteacherdescribingonestudentasinnocentisnot,inourview,enoughtoamounttoareport

thathewasacivilian.Ontheotherhand,whilethesearequiteclearlymediareportsofmilitant

deaths,wedonotknowwhatitmeanstobeayoungster(under18andthusachild?)whoisassociatedwiththeHaqqaniNetworkinWaziristantheextenttowhichtheassociationincludes

actualinvolvementinmilitantactivities,orbehaviorwhichwouldmakehimdirectlytargetableunder

unknownU.S.standards.

MediaReportsRelianceonUnnamedPakistaniGovernmentSources

Mediareportsprovideaweakbasisfordeterminingtheidentityofthosekilledbecausetheytypically

categorizethedeadasmilitantsbycitingunnamedPakistaniofficials,althoughinsomecasesthey

corroboratestatementsbyofficialswithstatementsbyanonymousresidents.Wedonotknowwhothe

unnamedPakistaniofficialsare,althoughobserversbelievetheyarePakistaniArmyofficials.What

definitiontheseofficialsusetocategorizeapersonasamilitantorcivilianisunknown.Nordoweknow

howthePakistaniArmyconfirmssuchdeathsorthequalityofinformationitisabletorelyon,giventhe

limitedaccessibilityofsomeofthetribalregionstoeventheArmy.CriticsofthePakistanimilitaryargue

thatitdoesnotconducton-the-groundinvestigationsbeforeissuingontheconditionofanonymity

announcementsthatthedeadaremilitants;thesecriticscontendthatitisplausibletheArmyhasa

politicalinterestincategorizingasmanyofthosekilledindronestrikesasmilitants.19

Beyondgovernmentsources,mediareportsrarelyprovideanyadditionalidentifyinginformationabout

thedeadthatwouldenablethereaderortrackingorganizationstoreachtheirownjudgmentonthe

matter.Ourreviewofthe2011dataforPakistanshowsthatoutof330to575militantsreporteddead,

forbetween100to219individualsidentifiedasmilitants,thesolesourcemediareportsprovideis

anonymousofficials.Inthecaseof36to71ofthosedead,absolutelynofurtheridentifyinginformation

isprovidedbeyondtheclaimthattheyweremilitantsorallegedmilitants.Fortheremaining64to148

dead,someidentifyinginformationisprovided,suchasanallegedconnectiontoaparticularmilitant

18

SeeColumbiaDataset,Strike46.19

See,e.g.,MuhammadIdreesAhmad,GunboatsandgurkhasintheAmericanImperium,

AlJazeera.com(July14,2012).


18/36

1


grouporleader.20

Oftentheadditionalidentifyinginformationissimplyaclaimthatthedeceasedwere

foreign,whichmightconflictwithotherreportsclaimingtheywerelocal.

Thetrendofheavyrelianceonanonymousofficialsisnotlimitedtothe2011dataset.NewYork

UniversitySchoolofLawsGlobalJusticeClinicandStanfordsInternationalHumanRightsandConflict

ResolutionClinicanalyzedthearticlesreliedonbytheNewAmericaFoundationforitsdataondrone

strikesinPakistanfromJanuarytoJuly2012.NYUsanalysisfoundthatin74percentofthearticles,the

onlysourceforthenumberofmilitantskilledwasanonymousgovernmentofficials(almostalways

unnamedPakistaniofficials).21

InconsistentReportingAmongMediaReportsandtheImplicationforTrackingOrganizationCounts

Inthemajorityofstrikereportsthatweanalyzed,diversedeathcountsarereportedamongdifferent

sources.Inconsistenciesastothenumberofdead,theidentityofthedead,theobjecttargeted,and

eventhelocationofastrikearenotuncommon.

Amongreputablenewsorganizations,thereportednumberofdeadoftendiverges.Sometimesdifferent

reportednumbersreflectthedeathcountchangingovertimeasmoreinformationbecomesavailable,

butmoreoftenitappearsthatfiguresaresimplyconflicting.Sometimesthedifferenceissignificant,but

evenwherethedifferencesaresmall,thetotalsaddup.Tovaryingextents,thetrackingorganizations

choicesofapoolofsourcesskewedtheircountsupwardordownward.

Ofthethreetrackingorganizationswhosedataweanalyzed,theBureauofInvestigativeJournalism

typicallycitesthelargestnumberofsources.TheBureauisfollowedbytheNewAmericaFoundation,

withtheLongWarJournaltypicallyrelyingonthefewestsources,oftenonlyoneortwoarticles.The

divergencesinmediareportsandvaryingmediareportpoolsizesisonereasonwhytheBureauhasthe

highestuppercasualtyfigures,followedbyNewAmericaandthenbytheLongWarJournal.22

Interms

ofindividualstrikes,therelianceonalargernumberofsourcesbytheBureauandNewAmericaoften

leadstotheincorporationoflower,aswellashigherfigures.Themostsignificantdifferencethatresults

howeverisamuchhigherupperfigure,andawidergulfbetweenupperandlowerfigures.

Thedivergencesinmediareportsandtherelationshipbetweenthenumberofsourcesandhigher

casualtyfiguresunderscoresthatmediatrackingorganizationsshoulduseawiderangeofcredibleand

reliablemediaincataloguingdeaths.Moreover,whereitisnotpracticallyfeasiblefortracking

20SeeCountsofMilitantsReportedKilled(Pakistan2011)(chart), infraat22.21

SeeNYUCenterforHumanRightsandGlobalJustice&StanfordInternationalHumanRightsClinic,

LivingUnderDrones43-54(2012);seealsoConorFriedersdorf,FlawedAnalysisofDroneStrikeDatais

MisleadingAmericans,TheAtlantic(Blog)(July18,2012),

http://www.theatlantic.com/politics/archive/2012/07/flawed-analysis-of-drone-strike-data-is-

misleading-americans/259836/ .22

SeeColumbiaDataset,Strike4,Strike11,Strike12,Strike15,Strike42,forexamplesofcasesinwhich

relianceonlimitedsourcesbysomeorganizationswasthecauseofvaryingcasualtycounts.


19/36


organizationstocollateallrelevantreports,theirstatisticsonreporteddeathsareincompleteand

shouldbeprovidedwithaqualificationthattheydonotrepresentallreports.

Moreover,relianceonEnglish-languagesourcessignificantlylimitsthenumberofreportsthatcanbe

accountedforbythetrackingorganizationsanalyzed,aswellasforourowncount.23

Thisisparticularly

relevantinacontextwherelocalmediaoutletsinnon-Englishspeakingregionsmaybemorewillingandabletoinvestigatetheparticularstrikes.BillRoggio,ManagingEditoroftheLongWarJournal,toldus

thatnecessaryrelianceonEnglish-languagesourcesisahindranceinhiswork.24

Afurtherlimitationofmediatrackingingeneralisthatreportingoneverysingleallegeddroneattack,

andfollowingupontheseattacksasdeathcountschangeovertime,isnotnecessarilythepriorityof

journalistsandmediaoutlets(particularlyEnglishlanguageasopposedtomorelocalones).In

particularlycontroversialorimportantcases,wherestrikeshaveallegedlykilledalargenumberof

civiliansorimportantmilitantleaders,mediacoverageismoreextensive.

UncountedStrikes

Diversereportingillustratesnotonlyhoweasyitcanbetomisssomereportedfigures,butitmayalso

bethecasethatentirestrikesgounreported,orthatreportsofstrikesareoverlooked.

TheBureauofInvestigativeJournalismlistsanumberofcaseswhichmayormaynotbedronestrikes,

forwhichasinglesourceisavailable.WhilethesestrikesarelistedintheBureausdata,thefiguresare

notincorporatedintotheBureauscasualtycount.25

Forexample,TheNews,areputablePakistani

source,reportedthatasecondstriketookplaceonAugust22,2011,killinganArabfamily.Itistheonly

identifiedsourcetohavereportedonthestrike.Inanothercase,UrduTVstationAjjreportedthata

strikeonSeptember4,2011killedupto7allegedmilitants. 26TheonlyidentifiedsourceisChineseoutletXinhua,whichregularlycitesforeignlanguagemediareportsinitsownEnglish-languagereporting.

27

Thedecisionnottocountthesesingle-sourcestrikesincasualtycountsisasensibleone,reflecting

concernaboutthefailureofthemediatomorewidelyreportthestrike.Wecannottellwhyparticular

strikesarenotmorewidelyreportedininternationalandPakistaniEnglish-languagemedia,andwhileit

maybeareflectiononthecredibilityofthereport,itmayalsobeareflectionontheprioritiesofthe

mediainanyparticularnewscycle,oronthedifficultiesofreportingfromsuchremoteregions.Assuch,

itmaywellbethatafamilywaskilledbyastrikeinAugust,andthattheirdeathshavegone

unaccountedforinthecasualtyfigures.

23

However,theBureauhascitedtonewsreportsinotherlanguages,includingUrdu.24

ColumbiaHumanRightsClinicphoneinterviewwithBillRoggio(July23,2012).25

BureauofInvestigativeJournalism,CovertUSStrikesinPakistan,YemenandSomaliaour

methodology,http://www.thebureauinvestigates.com/2011/08/10/pakistan-drone-strikes-the-

methodology2/(lastvisitedOctober9,2012).26

StrikeOb240cintheBureausdataset.27

StrikeOb240ciintheBureausdataset.


20/36

2


Thepotentialforentirestrikestobemissedisalsoillustratedbythedelayedreportingofastrikeon

October26,2011.ReportsofthestrikedidnotsurfaceuntilOctober28,withReutersreporting:

SecurityofficialsandTalibansourcessaidtheremotelocationofthesuspectedstrikepreventeditfrom

beingreportedearlier,andreportsofthestrikeonlysurfacedwhentheinjuredarrivedatahospitalin

MirAli.28

Thestrike,whichreportedlykilledbetween13-22people,includingaseniorTalibanmember,

hadbeenmissedentirelybytheNewAmericaFoundation,buttheyaddedittotheirdatasetafterwe

providedthemourfindings.29

Whileinsomecases,suchaswiththemissedOctober26strike,amorerigorousmethodologymay

improvetheestimatesoftrackingorganizations,apersistentproblemisthebiasesofmediasourceswe

describedabove.Takentogether,theseflawsmakeaggregationofmediareportsaninadequatemetric

fordeterminingthenumberandidentityofindividualskilled.Whileon-the-groundandin-depth

reportingarealwayslimitedinwarzones,weareconcernedthatthepublicandpolicymakersmayfailto

recognizethatestimatesofdronestrikesdeaths,whileprovidedingoodfaith,areincompleteandmay

failtoreflectthetruehumanitariancostofstrikes.

IV.OurRecountofDroneStrikeDeathsWeconductedanindependentreviewofthemediasourcesreliedonbytrackingorganizationsand

arrivedatdifferentcountsontheoverallnumberofindividualskilledandtheiridentities.The

discrepancieswedescribebetweenourcountandthatofthetrackingorganizationsarenotmeantto

impugntheeffortsoftheorganizations.Rather,ourrecountreflectstheextenttowhichcivilian

casualtieswhicharecrediblyreportedmaynonethelessbemissedbythetrackingorganizations;it

underscoresthatthepublicandpolicymakersdonothavecompleteinformationaboutthe

humanitariancostofdronestrikesandshouldnotusetheestimatestojustifyanexpansionor

continuationofdronestrikes.

Basedonaclosereviewofeachtrackingorganizationsdataandmediasources,wecametodifferent

countsofthetotalnumberofindividualskilled,thenumberofallegedcivilianskilled,andthenumberof

allegedmilitantskilled.Whenwecomparedeachofthetrackingorganizationsownoverallestimates

withtheirindividualstrikecounts,wefoundsomediscrepancies.Moreover,whenwereviewedthe

originalmediasourcesthetrackingorganizationsreliedupon,wecametodifferentcounts;ourcounts

weresimilartothatoftheBureauofInvestigativeJournalism,butourcountofreportedcivilian

casualtiesinparticularwassignificantlyhigherthanthecountsbyNewAmericaortheLongWarJournal.

Belowweexplainfurtherthecausesofdifferencesbetweenthethreetrackingorganizationsanalyzed,

andourownwork.Wehavealsoendeavoredtobeastransparentaspossibleaboutourmethodologyforeachindividualstrikeinourstrike-by-strikeanalysis.

28

JibranAhmad,SuspectedU.S.dronekills13PakistaniTaliban,Reuters,Oct.28,2011.29

ColumbiaDataset,Strike67.


21/36


ComparingTrackingOrganizationsOverallNumbersandStrike-by-StrikeTallies

Whenweconductedastrike-by-strikereviewbasedonthetrackingorganizationslistedcountsfor

Pakistan2011,weinsomecasescametodifferentoverallcountsofdeathsthantheorganizations

reportedastheiroverallestimates.Belowwelistourrecountoftheorganizationsinformation,with

theirtotalcountsinparentheses:

TheBureauofInvestigativeJournalismo Totalkilled:CountingtheBureausownfiguresonastrike-by-strikebasis,wecounteda

rangeof473to669totalindividualskilled(comparedtotheBureausowntotalcountof

473-663)

o Civilians:Arangeof68to157civilianskilled(theBureaureportsanupperrangeof154civilians)

NewAmericaFoundation:o Totalkilled:Arangeof366to599totalindividualskilled(consistentwithNewAmericas

owntally)o Militants:331to524militantskilled(NewAmericareportsarangeof336to535)o Civilians:Arangeof3to9civilianskilled(consistentwithNewAmericasowntally)o Unknown:Arangeof32to66unknownskilled(NewAmericareportsarangeof27to

58)

TheLongWarJournal:o Totalkilled:Arangeof438to478totalindividualskilled(LongWarJournalreports435

total)

o Militants:Arangeof421to475militantskilled(LongWarJournalreports405)o Civilians:Arangeof3to38civilianskilled(LongWarJournalreports30)

InthecaseoftheBureauofInvestigativeJournalismandtheNewAmericaFoundation,thedifferences

hereareveryminor,applyonlytothemaximumfigures,andarelikelytheresultofasmallcounting

errorinthecompilationofdata.30

ThedifferencesintheLongWarJournalstotalsaremoresignificant,

andarelikelytheresultoftheJournalslessextensiveformofreporting.WhereasNewAmericaandthe

Bureaulistastrike-by-strikecountontheirwebsite,ourstrike-by-strikefiguresfortheJournalcome

fromseparatearticlespublishedoneachstrike.BillRoggiooftheLongWarJournaltoldusthat

generally,theJournalstotalstatisticsreflectwhatispublishedinthearticles,butonoccasionthe

figuresmightbeupdatedwithoutpublishinganewarticle.Wefoundthatinsomecases,anarticles

conclusiononthenumberoridentityofthedeadwasambiguousourtallyoftheJournalsfigures

reflectsthatambiguity,whereastheJournalsownsingle-figuretotalsdonot.31

30

Wewerenotabletodeterminethesourceofthecountingerror.31

ThemostsignificantexampleisthestrikeonMarch17,2011,listedasstrike21inourdataset.The

Journalputsreporteddeathsatbetween14and35,andnotesthataccountsdifferedastotheidentity

ofthedeceased(aseithermilitantsorcivilians).Assuch,ourtallyfortheJournalis14-35totaldeaths,0-

35militants,and0-35civilians.Weconsiderthistobethemostaccurateportrayalofthecontentofthe


22/36

2


article,howeveritisclearlynotcommensuratewiththeJournalsowntotaltally,whichincludesonly

singlefigures,andinparticular,atotalciviliancasualtyfigureof30fortheentireyear.

0

100

200

300

400

500

600

700

800

Bureauof

Invesgave

Journalism

NewAmerica

Foundaon

LongWar

Journal

OurCount

Maximumcount

Minimumcount

TotalReportedKilled(Pakistan2011)

0

100

200

300

400

500

600

700

NewAmerica

Foundaon

LongWar

Journal

OurCount Militants

Strongly

Idenfied

Maximum

Minimum

CountsofMilitantsReportedKilled(Pakistan2011)

NB:TheBureauofInvesgaveJournalismdoesnotspecifically

trackamilitantcasualtycount


23/36


ComparingOurIndependentCountandCriteriatotheTrackingOrganizationsCounts

Afterrecalculatingthetrackingorganizationsestimatesbasedonastrike-by-strikereview,wereviewed

themediasourcesthetrackingorganizationsreliedupon,andcametoanindependentcountofindividualskilled:

o Total:456to661totalkilledo Allegedmilitants:330to575o Allegedcivilians:72to155

Usingtheminimumcasualtyfigures,wecounted2300percentmoreciviliancasualtiesthanwere

reportedbytheNewAmericaFoundationasciviliancasualties;wecounted140percentmore

civiliancasualtiesthanwerereportedbyNewAmericaFoundationaseithercivilianorunknown

casualties;wecounted140percentmoreciviliancasualtiesthantheLongWarJournalreported;andwe

counted5.9percentmoreciviliancasualtiesthantheBureauofInvestigativeJournalism,towhichour

countwassubstantiallysimilar.

Ourrange,andthediscrepancybetweenourestimatesandthatofthetrackingorganizations,partly

reflectsthedifficultofassessingtheidentityofthosekilledbasedonmediareports.Belowwesetout

ourcriteriaforthisassessment,butwerecognizethesearesubjectivedeterminationsandthatevery

countwillvary;thisinevitablevarianceunderscorestheinherentuncertaintyaboutdronestrikedeath

020

40

60

80

100

120

140

160

180

Bureauof

Investigative

Journalism

New

America

Foundation

LongWar

Journal

OurCount Civilians

Strongly

Identified

Maximum

Minimum

Counts ofCiviliansReportedKilled(Pakistan2011)


24/36

2


estimatesbasedonmediareports,andthattheU.S.governmenthasultimateresponsibilityfor

accountingforciviliancasualtiesandprovidinganofficialestimate.

CriteriaforIdentifyingCiviliansandMilitantinEstimates

Aswehavenoted,toooftenthemediareportsonwhichtrackingorganizationsbasetheircasualty

countsareweak,bothbecauseoftheambiguityofthetermsmilitantandcivilianandthelimitationsofthemediareportsprimarysources.Wereviewedthe2011mediareportscitedbytracking

organizationscloselytodeterminetherelativestrengthoftheidentificationsofmilitantsand

civilians.Ourownanalysisiscircumscribedbytheambiguityoftheseterms,yetwehavetriedto

isolaterelativestrengthsandweaknessesbasedonthefollowingcriteria:

Strongidentificationwherethedeceasedareindividuallyidentifiedbyname,and/orwherethereportedidentificationofthedeceasediscorroboratedbyanindependentinvestigation(the

on-the-groundinvestigationconductedbytheAssociatedPressandpublishedinFebruary2012,

andtheprimaryinvestigationsoftheBureauofInvestigativeJournalismsresearchersin

Waziristan.)

Mediumidentificationwheretherearemultipleoriginalsourcesfortheidentificationofthedead.Forexample,bothanonymousofficialsandalocalresident.

Weakidentificationwherethereisonlyonesourcefortheidentification.Forthispurposewetreatmultipleanonymousofficialsasonesource,andpluralunnamedresidentsasonesource.

Whilewehavetreatedidentificationbynameasastrongidentification,notallsuchcasesareequal.In

somecases,thenameofanotherwiseunknownpersonisallthatisreported.Inothercases,suchasthe

March17strike,theidentificationofciviliansbothbynameandbyanunusuallywidearrayoforiginal

sources,astrongnarrativeexplainingthecircumstancesofhowsomanyciviliansapparentlycametobe

attacked,andreportingofmassprotestsagainstciviliandeathsasaresultoftheattack,provideforan

evenmoreconvincingidentificationofciviliandeaths,particularlyinabsenceofareasonablecounter-

narrative.

Whilemostnamedmilitantsareleaders,thenamedmilitantskilledalsoincludethethreenamed

youngstersfromtheJuly12strikediscussedabove.Eventhestrongestmilitantidentifications,

identifyingnamedandknownmilitantleaders,arefallible.Thishasbeenproveninnumerousinstances

wherethereporteddeadhavelatersurfacedaliveandwell.32

32

AnotableexampleisHakeemullahMehsud,theleaderofthePakistaniTalibanwhowasincorrectly

reporteddeadonanumberofoccasions.SeeBillRoggio,HakeemullahMehsudnotdeadyet,The

LongWarJournal,April29,2010

http://www.longwarjournal.org/archives/2010/04/a_senior_pakistani_i.php.SeealsoBillRoggio,

HakeemullahMehsudrumoredkilledinrecentdronestrike,TheLongWarJournal,January15,2012,

availableathttp://www.longwarjournal.org/archives/2012/01/hameemullah_mehsud_r.php.


25/36


Recognizingthatsuchcategorizationsaremoreanartthanascience,wehavetriedtobeastransparent

aspossibleastohowwehavecategorizedeachstrikeinourstrike-by-strikeanalysis.Wehavealso

brokensomeofthefiguresdownintomoreobjectivesubparts33

:

StronglyIdentifiedMilitants:Total105outof330-575(32%oftheminimummilitantcount)o 34areidentifiedbynamethedeathsof23areunconfirmed,whilethereissomelevel

ofconfirmationfortheremaining11deaths

o 71ofthedeceasedsreportedidentificationasamilitantiscorroboratedbyanindependenton-the-groundinvestigation

StronglyIdentifiedCivilians:Total56outof72-155(78%oftheminimumciviliancount)o 52areidentifiedbynameo 16ofthedeceasedsreportedidentificationasacivilianiscorroboratedbyan

independenton-the-groundinvestigation

o Thisincludes12individualswhoarebothidentifiedbynameandwhosecivilianidentityiscorroboratedbyindependentinvestigation

WeaklyIdentifiedMilitants:Total113-232outof330-575o 100-219areidentifiedasmilitantssolelybyanonymousofficials.Ofthese:

Absolutelynoidentifyinginformationisprovidedfor36-71ofthedeceased Someidentifyinginformationisprovidedfortheremaining64-148(forexample,

thattheywereforeign,alinktoaparticularmilitantgroup)

o 13areidentifiedbyadifferentsingleoriginalsource,ortheoriginalsourceisnotapparentinreports

WeaklyIdentifiedCivilians:Total2-73outof72-155o 2-12areidentifiedascivilianssolelybytheBureauofInvestigativeJournalismsprimary

research,incaseswherethereareconflictingaccounts

o 0-53areidentifiedascivilianssolelybyaresidentorlocalsourceso 0-8areidentifiedascivilianssolelybyananonymousofficial

Thesebreakdownsarerepresentedinthegraphsbelow,asaportionoftheminimumfiguresfortotal

militantsandtotalcivilianskilled.Strongidentificationsarerepresentedinred,andweak

identificationsarerepresentedinblue.

33

Ourdatasetandouranalysisofthestrengthofexistingreportsdoesnotincorporatetheinvestigationsreported

inNYUCenterforHumanRightsandGlobalJustice&StanfordInternationalHumanRightsClinic,LivingUnder

Drones57-66(2012),availableatwww.livingunderdrones.org.Thisreport,releasedafterthefinalizationofour

data,providesfurtherrelevanttestimonyrelatingtothestrikesonMarch17andJune15,andiscitedbythe

BureauofInvestigativeJournalism.


26/36

2


Solesource:anonymous

officials,noidenfying

informaon

Solesource:anonymous

officials,someidenfying

informaon

Othersinglesource

idenficaonorsource

unclear

Idenfiedbyname,death

unconfirmed

Idenfiedbyname,death

confirmed

TotalMilitantsKilled:StrengthofIdenficaon

Basedonminimumcasualtyfigures

TotalCiviliansKilled:StrengthofIdenificaon

Idenfiedbyname

Bothidenfiedbynameand

reportscorroboratedby

independentinvesgaon

Reportedidenty

corroboratedbyindependent

invesgaon

Weakidenficaon:basedon

Bureauresearchbut

conflicngwithotherreports

OtherBasedonminimumcasualtyfigures


27/36


V.ComparisonoftheTrackingOrganizationEstimatesWecomparedthecasualtytrackingconductedbytheBureauofInvestigativeJournalism,theNew

AmericaFoundation,andtheLongWarJournal,fordronestrikesinPakistanin2011,withtheprimary

motivationofunderstandingthequitestarkdifferencesbetweeneachorganizationsresults.The

influencethesetrackingorganizationshavemakesitcrucialtounderstandtheirmethodologiesandthe

reasonswhytheyarriveatdifferentestimates.Wheretrackingorganizationssignificantlyunderestimate

thenumberofcivilianskilled,theymaydistortourperceptionsandprovidefalsejustificationto

policymakerswhowanttoexpanddronestrikestonewlocations,andagainstnewgroups.

Wegaveeachorganizationanopportunitytorespondonastrike-by-strikebasisinthosecaseswhere

ourconclusiononcasualtyfiguresdifferedfromtheirs.Asnotedabove,theNewAmericaFoundation

declinedtospeakwithus,butdidsignificantlychangetheirdataafterreceivingourcritiques.34

Bill

RoggiooftheLongWarJournalspoketousandprovidedmanyinsightsintohiswork,butdidnot

responddirectlytoourstrike-by-strikedifferences.TheBureauofInvestigativeJournalismdidrespond

andengagedinadialoguewithusonparticularstrikes.Inanumberofcases,theBureausfeedback

causedustorevisitouranalysis,ortheBureauagreedwithusanddecidedtochangetheirowndatato

takeaccountofpointsthatweraised.

TrackingCivilianCasualties

Incomparingeachorganizationsworkwithregardstociviliancasualtiesinparticular,itisimportantto

notethattheBureauofInvestigativeJournalismistheonlyorganizationtohaveconsistentlyand

activelytrackedciviliancasualties.

WhiletheLongWarJournaldoesprovideaciviliancasualtycount,BillRoggionotedthatthisisnotthe

Journalsprimaryfocus.RatherthefocusandexpertiseoftheLongWarJournalisinpatternsof

operationsthattheU.S.isconductingthisprogram,whotheyretargeting,wheretheyretargeting,

whoseareasarebeingtargeted,whichTalibancommanderhaspurviewoverthoseareas.Thisis

clearlyreflectedintheJournalsworkunliketheBureauandNewAmerica,theLongWarJournaldoes

notpublishastrike-by-strikelistofcasualties,whichmakestheirtotalcasualtycountsomewhatless

transparent.Itdoeshoweverpublishseparatearticlestrackingeachstrike,inwhichagreatdealof

contextualinformationisprovidedtohelpthereaderunderstandtheplaceofeachstrikewithinwider

U.S.operations.TheJournalalsodoesexceptionalworkintrackingtheallegeddeathsofmilitant

leaders.

UntillateAugust2012,theNewAmericaFoundationsdatasetdidnotincludeacategoryforcivilians

andtrackedonlythedeathsofmilitants,militantleaders,andothers.Theothercategorywas

ambiguousandNewAmericascalculusforitwasunclear;inallofthePakistan2011strikeswherethe

NewAmericaFoundationhadcountedotherdeaths,wefoundthattherewereinfactreportedcivilian

deaths.InlateAugust2012,NewAmericareplacedtheothercategorywithcivilianandunknown

categories.Thisisapositivedevelopment,makingthedatasetmoretransparent.AccordingtoNew

34

WhetherthiswasacontributingcausetoNewAmericaschangesisunknown.


28/36

2


America,theunknowncategoryisusedifthevariousmediareportsaresocontradictorythat[New

Americais]notcomfortabledrawingaconclusion.35

NewAmericasnewcivilianestimateissurprisinglylowcomparedtoitsoldothercountalow-

rangefigureofonly3civilianskilledcomparedtoalow-rangefigure16otherskilledinPakistanin

2011.AlsoperplexingisthatNewAmericasnewciviliancountisonlyonepercentofitstotaldeathcountfortheyear,whereasithaduntilrecentlyclaimedaciviliandeathratein2011of7percent.

36The

shiftingclassificationofcasualtiesandtheambiguousdataunderscorethelackofstandardizationacross

trackingorganizationcounts,andthefallibilityoftheseestimatesasagaugeonthetruenumberof

civiliandeaths.

ReasonsforDiscrepanciesinTrackingOrganizationsCasualtyEstimates

1.Thenumberofmediasourcesreliedon

Asdiscussedabove,thewidertherangeofcrediblemediasourcesreliedupon,thewidertherecorded

reportedcasualtyfigureswillbe.Inmanycases,oneorganizationhasalowerminimumand/orhighermaximumcasualtyfigurethantheotherssimplybecauseitcitedtoanadditionalarticleprovidingthat

higherorlowernumber.Asdiscussedabove,thediversityandinconsistencyinthereportingondrone

casualtiesinPakistanmakestheentirebusinessofmediatrackingdifficult,asthereisalwaysthe

possibilitythatanothercrediblearticlewithanagaindifferentcasualtyfigureisoutthere.Wehave

pointedouttotrackingorganizationswheretheyaremissinganadditionalsourcewhichwouldjustifya

lowerorhighercasualtyfigure.TheBureauofInvestigativeJournalismwastheonlyorganizationto

respondtothisinformation,includingbyincorporatingouradditionalsourceandupdatingtheirfigures.

Inourview,thewiderthenumberofcrediblesourcesreliedupon,themoreaccuratethecasualtycount

isasafigurewhichisintendedtoreflectreportsofdeaths.Inlightofthesparsecontentandlimitedsourcesreliedoninreports,thereisrarelyenoughinformationforatrackingorganizationtobeableto

discardareport,exceptinthecaseofupdatedfigures.

2.TheBureausnon-mediasources

TheBureauhasundertakentwofieldinvestigationsintopossibleciviliancasualtiesinPakistan.This

primaryresearchisoneoftheBureaussourcesforitscasualtyfigures,butisnotincorporatedbyeither

theNewAmericaFoundationortheLongWarJournal.ThefailuretotaketheBureausownresearch

intoaccountisonecauseofdiscrepanciesbetweentheBureausfiguresontheonehand,andNew

AmericaandtheJournalsontheother.Interestingly,theonlyorganizationtociteanyoftheothers

workistheBureau,whichoccasionallycitestoLongWarJournalarticles.

35

NewAmericaFoundation,TheYearoftheDrone:Methodologyavailableat

http://counterterrorism.newamerica.net/drones/methodology.36

PeterBergenandJenniferRowland,CIAdronewarinPakistaninsharpdecline,CNN,Mar.28,2012,

availableathttp://edition.cnn.com/2012/03/27/opinion/bergen-drone-

decline/index.html?iref=allsearch.


29/36


TheBureausresearchersundertookfieldinvestigationsinWaziristan,wherealmostalldronestrikesin

Pakistanin2011tookplace.Journalistsdonottypicallyhaveaccesstothearea,andvirtuallynomedia

reportsarebasedoninformationgatheredinsidetheregionfirsthandorareabletobeverifiedbythe

mediaorganizationitself.Assuch,theBureauson-the-groundresearchisparticularlyusefulasatoolfor

corroboratinginformationpresentedinmediareports.

InlightoftheBureausaccesstotheregion,theirprimaryresearchisinourviewatleastasreliableand

credibleasthestandardmediareportswhichallthreeorganizationsrelyon,ifnotsignificantlymoreso.

Inanumberofcases,theBureausresearchershavealsobeenabletoprovideawealthofinformation

regardingtheidentityofthedead,includingnamesandtribes.Thislevelofinformationisrareinmedia

reportsexceptincaseswhereanimportantmilitantleaderhasallegedlybeenkilled.

Inlightoftheabove,wehaveincludedtheBureausprimaryresearchinourowncount.Asdiscussed

above,heretheBureausresearchcorroboratesthereportedidentityofthedead,wehavetreatedthat

identificationasparticularlystrong.

TheBureauhasalsocitedothernon-mediasourcesthatarenotincorporatedbytheNewAmerica

FoundationortheLongWarJournal,suchaslegaldocumentsfiledonbehalfoftheallegedcivilian

victimsofdroneattacks.Suchdocumentsaresometimesabletoprovidemoredetailedinformation,

includingthenamesofthedeceased,thanisavailableinmediareports.Whileitmightbearguedthat

thesedocumentsarebiased,havingbeenfiledonbehalfofapartytolitigation,theysometimesprovide

additionalcorroboratingdetailssometimes,farmoreinformationthanprovidedinmediareportsthat

citeunnamedanresidentorofficial.

3.Theproblemofupdatingfigures

Itisnotunusualfordeathcountstochangeovertime.Thisistobeexpectedinanyinstanceofbombing,

asmoreinformationbecomesavailable,asmorebodiesarepulledfromtherubble,orasinitialsurvivors

diefrominjuries.

Giventhecommoninconsistenciesinthereportingondronecasualties,andthefactthateachstrikeis

notcommonlyrevisitedwithanumberofupdatedarticlesbythesamemediaoutlet,itisoftenvery

difficulttodeterminewhetherdifferencesinreportingindicatethatlaterinformationhasbeenupdated,

orwhetherthedifferencesarejustanotherinstanceofinconsistentreports.Giventhedifficultyof

determiningthisissue,wehavetriedtoapproachitconservatively.

Theorganizationsanalyzedappeartotreatthisissuedifferently,whichisoneofthecausesofthe

discrepanciesbetweentheirfigures.Inanumberofcaseswehavealsoapproachedtheissuedifferently

totheorganizationsanalyzed,leadingtosomediscrepanciesbetweenourowncountandtheirs.Dueto

alackofclarityinthereporting,thereissomeroomforreasonabledisagreementonhowthismatteris

treated.Infairnesstotheorganizationsanalyzed,wealsohavetheadvantageofcomingtothisexercise

atalaterpointintime.Weappreciatethatmediatrackingofcasualtiesisanextremelytime-intensive,

andonemaynothavetheresourcestotrackallnewstrikeswhilecontinuingtorevisitalloftheold

ones.


30/36

3


Eachsuchincidentisdiscussedonacase-by-casebasisinourstrike-by-strikeanalysis,butingeneral

inconsistentupdatingaddsanotherlayerofambiguitytothedatacurrentlyinformingpublicdebateon

drones.Ingeneral,however,theBureauofInvestigativeJournalismappearstoupdateitscountsbased

onlaterreportsmoreconsistentlythantheotherorganizations.Insomecases,theBureauhastreated

earlierfiguresassupersededwherewehaveconservativelydecidedtocontinuetotreatthosefiguresas

relevant.37

TheNewAmericaFoundationandtheLongWarJournalhaveinsomecasescontinuedtousefigures

which,inourview,havebeensuperseded.BillRoggiotoldusthatoccasionally,theLongWarJournal

mightupdateitstotalfigureswithoutpublishinganewarticleonthestrikewhichthenewfigurecomes

from.Asdiscussedabove,wehavebasedouranalysisoftheLongWarJournalsfiguresonitsarticles,

andforthatreasonwecannotbecompletelysurethatanyindividualfigurehasnotbeenupdated.Ina

numberofcases,theLongWarJournalsarticleshavecitedfigureswhichwebelievehaveclearlybeen

superseded.38

Forexample,forastrikeinNorthWaziristanonJune15,2011,theLongWarJournalreliesonanow

deadlinktoDawnforitscountof5militantskilled.HoweverDawnreportedagainonthestrikeonJune

17,identifying4civilianswhowerereportedlykilled.

PriortoitsrecentdataoverhaulinAugust2012,afailuretoupdateitsfigureswasalsoamajorproblem

withtheNewAmericaFoundationscasualtycount.TheBureauofInvestigativeJournalismdescribed

NewAmericasdataduringthistimeasbeingofasnapshotnaturethatonlytakesintoaccountafew

articlesonthedayofthestrike.39

Sinceupdatingitsdataanditswebsite,NewAmericahasmade

significantimprovementsinthisrespect.40

Howeverdespiteupdatingitsdata,thereareafewinstances

inwhichNewAmericacontinuestousefigureswhichwebelievehavebeensuperseded.41

37

Seeforexample,intheColumbiaDataset:Strike10,February20,2011;Strike18,March13,2011;Strike36,June8,2011;Strike41,June27,2011;Strike58,September11,2011;Strike74,November16,

2011.38

SeeforexampleintheColumbiaDataset:Strike11,February21,2011;Strike21,March17,2011;

Strike24,May6,2011;Strike73,November15,2011.39

ChrisWoodsAnalysis:CNNexpertsciviliandronedeathnumbersdontaddupTheBureauof

InvestigativeJournalism,July17,2012,availableat

http://www.thebureauinvestigates.com/2012/07/17/analysis-cnn-experts-civilian-drone-death-

numbers-dont-add-up/40

ButseeNYUCenterforHumanRightsandGlobalJustice&StanfordInternationalHumanRightsClinic,

LivingUnderDrones48(statingthatNewAmericaFoundationsstrikedatadonotappeartobe

updatedregularlytoincludethemostup-to-dateinformationaboutthenumberandidentitiesofvictimskilledindronestrikes). 41

SeeintheColumbiaDataset:Strike11,February21,2011;seealsoStrike21,March17,2011.Itis

puzzlingthatNewAmericadoesnotincorporatetheAPsindependentinvestigationfromFebruary2012

initslistofsources,althoughAssociatedPresssfigurefitsroughlywithinNewAmericasrange.Asone

oftheveryfewinstancesofonthegroundinvestigation,AssociatedPresssreportis,inourview,a

particularlyvaluableandcrediblesource.InaCNNarticlebeforeitsdataupdate,PeterBergen

comparedNewAmericasfigurestoAssociatedPresssacuriousapproachforamediatracking

organizationwhichonemightexpecttoincorporatesuchreports.SeePeterBergenandJennifer


31/36


4.DisregardingSources

Insomecases,theorganizationsfiguresdonotreflecttherangereportedinitsownlistedsources.

Wherelowerfiguresinarangearedisregardedbytheorganizationcitingthem,weassumethisis

becausetheorganizationcitingthemhastreatedthosefiguresassuperseded,amatterwhichweaddressabove.Disregardingupperfiguresishowever,inourview,slightlyhardertoexplain.

Insomecasesitmaysimplybeanerror.Thiswasthecasewiththeoneinstanceinwhichwefoundthe

Bureauhadnotincorporatedtheupperfigurefromoneofitslistedsources,asimplemistakewhichthe

Bureauthenrectified.

ThefailuretoincludesomefiguresishoweverdeliberateinthecaseoftheLongWarJournal,which

doesnotcountarangeofcasualtyfigures,withanupperandlowercount,butratherprovidesasingle

figure.BillRoggioexplainedtousthatwherethereisaninconsistencybetweenthesourcesthathe

cites,hetakesthefigurethathebelievesisthemostreliable.42

InsomeinstancestheNewAmericaFoundationhasalsofailedtoincorporateanupperfigurereported

initssources.43

SinceNewAmericaordinarilyreportsarangeofmediafigures,thereasonforthese

omissionsisnotalwaysclear.NewAmericaindicatesinitsmethodologythatitwillonlycountreported

civiliandeathsiftheyarereportedinmorethanonemediasource.44

Forreasonsdiscussedabove,inour

viewthisisnotappropriate.Asreportsofdronestrikesarebasedonsofeworiginalsources,the

reportingiserratic,andthebasisforidentificationisgenerallyextremelyweak,itisnotnecessarily

sensibletotreataclaimasreliablebasedsolelyonthenumberofoutletsitisrepeatedin.Forexample,

whilethesameofficialsmaybequotedinthewires,perhapsonlyonepaperwillquotearesidentor

afamilymember.Asofficialsarethemostcommonlycitedsource,thisalsoskewssources.For

example,inregardtotheMarch17strike,NewAmericalistsonlyavarietyofofficialsastheoriginal

source,whereasourownanalysisidentifiesanexceptionallylargeanddiversearrayofsources.45

Rowland,Civiliancasualtiesplummetindronestrikes,CNN,July14,2012,availableat

http://edition.cnn.com/2012/07/13/opinion/bergen-civilian-casualties/index.html. 42

Mr.RoggiogaveoneexampleofXinhuareporting11militantskilled,andAgenceFrancePresse(AFP)

reporting12militantskilled.AstheAFPreportwaspublishedalittlelaterandhetrustedAFPs

informationmore,heusedtheAFPnumber.ColumbiaHumanRightsClinicphoneinterviewwithBill

Roggio(July23,2012).43

SeeforexampleintheColumbiaDataset:Strike7,January23,2011;Strike28,May16,2011(failure

toincorporateCNNsupperfigureof10,althoughperhapstreatingitasatotalfortwostrikesontheday).44

NewAmericaFoundation,TheYearoftheDrone:Methodologyavailableat

http://counterterrorism.newamerica.net/drones/methodology45

Including:anonymousofficials;namedofficials(includingArmyChiefofStaffGen.AshfaqKayani,

AmbassadorHussainHaqqani,ForeignSecretarySalmanBashir,GovernorMasoodKasur,PrimeMinister

Gilani,MemberofNationalAssemblyKamranKhan);theTaliban;unnamedandanonymous

locals/tribesmen;namedlocals/tribesmen(farmerGulAhmed,SamiullahKhan);Bureauinvestigation;

legaldocumentsandcourtproceedingslodgedonbehalfofvictims.


32/36

3


Inanotherexample,NewAmericahadinitiallyincludedanumberofreferencestothereputedAfghan

publicationPajhwok.WhilePajhwokhadreportedciviliancasualties,NewAmericahadnotincorporated

theseclaimsintoitsdata.Inthemostrecentversionofthedataset,allofthereferencestoPajhwok

havesimplybeendeleted.

Moreover,insomecaseswheretherearemultiplestrikesononeday,NewAmericaappearstotreatthe

factthatsomemediaoutletshavenotreportedonallofthestrikesasevidencethatnotallofthestrikes

occurred.Forexample,fortwostrikesonJuly11,2011,mediareportsindicatethatatleast5people

werekilledinonestrike,andatleast10inanotherstrike.However,NewAmericatakesaminimum

countof7forbothstrikestogether,basedonaBBCreport.TheBBCreport,though,refersonlytoone

ofthetwostrikes:anattackonavehicle,andnotanadditionalattackonacompound.Assuch,New

Americasdataeitherintentionallyormistakenlyconflatesthetwostrikes,withtheeffectthatits

minimumcasualtycountmissesanentirestrike.IfNewAmericasomissionwasdeliberateandbasedon

theBBCsfailuretoreportthesecondstrike,itwouldnotinourviewbeappropriate;mostindividual

mediaoutletsdonotreportoneverysinglestrike.

Inourview,accountingforarangeofcasualtyfigures,asisdonebytheBureauofInvestigative

JournalismandgenerallybytheNewAmericaFoundation,isamoreaccurateandinformativewayof

trackingreportedcasualtiesthantheLongWarJournalsapproach.Whatisbeingtrackedisreportsof

deaths,andwherecrediblereportspresentdiverseresultsthatshouldbereflected.

Ifupperorlowerfiguresarediscardedandsourcesareusedselectivelyforanyreasonotherthanthe

existenceofupdatedinformation,thesereasonsneedtobemadeclearandexplicitinthedataandwith

thefinalcounts,aswhatisbeingcountedisthensomethingmorerefinedthanreporteddeathswhich

ishowthestatisticsarepresented.

4. NewAmericaFoundationsunknownandciviliancategoriesAsnotedabove,NewAmericaincludesanunknowncategoryincaseswherethereareconflicting

reportsabouttheidentityofthedead.Thisisarecentdevelopmentthatwasaddedafterwehad

completedourown2011analysis.Ourownapproachtoconflictingreportshasbeentotakealower

figureof0inboththecivilianandmilitantcounts.

Inourview,NewAmericasapproachinthisrespectisareasonableandmethodologicallysoundone.

Howeverinpractice,ourviewisthattheuseoftheunknowncategoryintheMarch17strikehas

unrealisticallyskewedtheciviliandeathcountdownwards.Althoughthenumbersaredifficultto

preciselydetermineinthatcase,reportsare,inourview,clearthatalargenumberofciviliansandsome

militantsdied.Inourview,24civiliansfromthisstrikealoneareparticularlystronglyidentified.

TheuseoftheunknowncategoryalsositssomewhatuncomfortablyalongsideNewAmericas

methodologyofnotcountingreportedciviliandeathsunlesstheyarefoundinmorethanone(English

language)newsoutlet.Assuch,whiletheunknowncategoryisusedincaseswherewewould

categorizetheidentificationofciviliansasstrong,anumberofreportsofciviliandeathsarenot

accountedforineithertheupper-civiliancasualtyfigureortheunknowncount.Thisisamatterof


33/36


methodologicaldifferenceinopinion,butinourviewthefinalresultisnotanentirelyaccurateportrayal

ofwhatispresentedinthemedia.46

ProblematicCivilianCasualtyEstimates

Ourclosereviewofthedatasuggestsmethodologicalproblemsintheciviliancasualtytrackingofthe

NewAmericaFoundationandtheLongWarJournal.Whileinmanycasesthereisroomforreasonable

disagreementonthemethodologicalapproachtaken,thetotaleffectofnumerousproblemswefound

resultsinsignificantundercountingofciviliancasualtiesinparticular.Asdiscussed,sincethepurported

focusofLongWarJournalisnotciviliancasualtytracking,thisisperhapsnotsurprising.

Theseverityoftheundercountisapparentwhenwecompareourfindingsofstronglyidentified

civiliandeathstoNewAmericaandLongWarJournalsfindingsofciviliandeathsgenerally.Wehave

assessed52ofthereporteddeadasstronglyidentifiedascivilians;theyareidentifiedbynameinmedia

reports.Ourfigureissignificantlyhigherthanthe totalciviliandeathcountsofNewAmericaandthe

LongWarJournal,at3-9and30,respectively.TheirfiguresdonotevencapturetherangeofreportedciviliandeadinonesinglestrikeonMarch17,2011oneforwhichthereisanexceptionallystrongcase

ofnumerousciviliandeaths.

Interestingly,thesameseverityofundercountingcomparedtostrengthofreportingisnotapparentin

themilitantcasualtycounts.Themilitantcasualtycountsrelyheavilyonwhatwehaveassessedasvery

weakreportedidentifications.NewAmericaandtheLongWarJournalslowermilitantcountishigher

thanourown,andsignificantlyhigherthanourcountofstronglyidentifiedmilitants.Whilethelatteris

tobeexpected,itcontrastssharplywiththecomparisontoourciviliancasualtystatistics.

46

SeeforexampletheNewAmericaFoundationsdataandreportingofthefollowingstrikes(allin

2011):7JanuaryGharLaley;21FebruarySpalga;8MarchLandidog;27JuneMantoi;1AugustAzam

Warsak;16AugustMiranshah.


34/36

3


VI.U.S.GovernmentsResponsibilitytoAccountforDrone

StrikeDeathsThediscrepanciesincountsbythetrackingorganizationscredibleandwell-resourcedinstitutions

underscorethedifficultyofgaininganaccurateunderstandingoftheimpactofdronestrikesfrommedia

reportsalone.Thepublicandsomepolicymakersarecompelledtorelyontheseestimatestojudgethe

impactofdronestrikesbecausetheU.S.governmenthasnotofficiallyprovidedanyinformationon

dronestrikedeaths.Whiletoutingthesuccessofthedroneprogramandparticularhigh-profilestrikes,

U.S.officialshaveavoidedprovidingspecificsandcitednationalsecurity.Indeed,althoughithas

acknowledgedoperationsinPakistanandYemenasageneralmatter,ithasrefusedtoofficially

acknowledgetheexistenceofitsdronesprogramincourtoropensessionsofCongressforeclosing

effectivelitigationandpreventinginformedpublicdebate.47

Ontheotherhand,theCIAhasaggressively

fendedoffcriticismthroughanonymousleakstothepressaforuminwhichitsclaimscannotbe

activelyquestioned.

ThepublichasnoinformationonhowandwhethertheU.S.tracksandinvestigatespotentialcivilian

deaths.Yet,inotherconflictsettingssuchasAfghanistan,U.S.officialshaveprovidedsomeofthis

information,withoutcompromisingU.S.security.48

AstheU.S.andothergo

Documents

COLUMBIA LAW SCHOOL - Human Rights Clinic: COUNTING DRONE STRIKE DEATHS