Judging Intelligence Success and Failure (Kringen)

7/31/2019 Judging Intelligence Success and Failure (Kringen)

1/37

Getting Beyond

She Said and He Said

1


2/37

The Premise There has been insufficient public discussion about

the appropriate benchmarks for evaluating the

performance of the intelligence community. The consequence is that public commentary tends to

reflect efforts to spin the latest developments.

The result is a debate that is largely reactive and

lacking in perspective.

2


3/37

How Did the IntelligenceCommunity Perform?

3


4/37

Apparently perspective matters

4


5/37

Senator Feinstein

SSCI Chair Our intelligence, and I

see it all, is way behind

the times. It is

inadequate.

The IC provided her

nothing that we didnt

read in the newspapers.

Lack of human

intelligence assets and

failure to use open source.

5


6/37

Representative Rogers

HPSCI Chair Intelligence clearly helps

us understand

developments in places

like Egypt, but it is not a

crystal ball. We have got

to be realistic about its

limits, especially

regarding the complex and

interactive behavior of

millions of people.

6


7/37

First By What It is Not

7


8/37

An intelligence failure is not: A matter of leaving

intelligence consumersin the suburb of

Surprise.

The world is anuncertainplace: Hewho lives by crystal ball

soon learns to eat groundglass. Edgar R.Fieldler

8


9/37

Intelligence failure is not: A development that

results in awkwardnessor embarrassment.

Intelligence failure

implies serious long-

term consequences.

9


10/37

Intelligence failure is not: An inability to keep up with the

latest reporting.

Director of NationalIntelligence Not Briefed onLondon Arrests BeforeInterview ABC News

10


11/37

And, it probably cant do

better than . . .

The forecasts of the individuals directly involved in

driving the situation (e.g. Russia-Georgia conflict of 2008) The ability of organizations to make estimates regarding

the outcomes of multi-year planning in their own

organizations (e.g, timelines for weapons development

and procurement). The adequacy of the base information available.

11


12/37

Its All About Risk Management

12


13/37

Understanding

the Dimensions of Risk Policy risk (e.g., the UBL

operation)

Political risk (operation to

rescue US hostages in

Iran)

Human risk ( Camp

Chapman)

Operational risk (e.g,

counterintelligence)

Investment risk (e.g., UAV

captured in Iran)

Lack of Investment risk

(principally, the failure to

collect criticalintelligence)

13


14/37

Lessons Learned: 2009 Attack

on CIA Base in Afghanistan Enforce greater discipline

in communications

Apply the skills and

experience of senior

officers more effectively

Require greater

standardization of securityprocedures

More carefully manage

information sharing with

other intelligence services

Strengthen our attention to

counterintelligence

concerns

Maintain our highoperational tempo against

terrorist targets

14


15/37

Fire-fighting: Common

Preincident Intelligence Failures The first and important

failure is in identifying

occupancy types and their

associated hazards.

The second common

failure is not becoming

oriented with the facility.

A third common failure is

improper information

management.

A fourth common failure

is not participating in

exercises.

The fifth common failure

is lack of information

maintenance.

The sixth common failureis not applying the lessons

learned.

15


16/37

16


17/37

How difficult was the intelligence problem?

Was there a coherent and focused intelligence

strategy?Did the alleged failure have significant and

lasting policy consequences?

Did the intelligence community execute theintelligence priorities provided by theexecutive and legislative branches ?

17


18/37

Key Questions: Part DeuxDid the assessment appropriately

characterize the key uncertainties?

How effective was the intelligence

community in highlighting areas of

significant concern?

Did the community provide insight into

potential policy options?

18


19/37

Benchmarks from the

Business World (A)ccurate forecasting in

the economic and businessworld is usually not

possible.

The future is never exactlylike the past.

It does not seem that theseriousness of the (2008)financial crisis, or the

resulting economicrecession, was predicted bythe great majority offorecasters.

The difficulty for decisionand policy makers . . . is tounderstand and accept theextent of future uncertainty

so that appropriate plans . . .can be formulated.

Source: Forecasting anduncertainty in the economic

and business world,International Institute ofForecasters, 2009

19


20/37

Benchmarks from the

Medical Profession Although expert witnesses

greatly determine theoutcomes of medical

malpractice claims, thereliability of experttestimony is frail. (Source:Poor Agreement AmongExpert Witnesses in BialDuct Injury MalpracticeLitigation: An ExpertPanel Survey Annals ofSurgery, November 2008

20


21/37

Key Standard: Recognizing

the Degree of Difficulty Not All Intelligence

Challenges are the Same

Recall the Belgian military

intelligence officer who was

only wrong twice

21


22/37

How Hard is the

Intelligence Challenge?Less Hard More Hard

Matters of fact or history

Does Iraq Have WMD?

Where is the Federal

Directorate of Supply and

Procurement?

Is Kim Chong-il Alive?

Developments based on a longand consistent track record

Decisions that are made

transparently

The outcomes of decisions that

have not yet been made or are

opaque.

Georgian military plans in

2008

Black Swans (e.g.,

discontinuous change) The outcomes of political-

social mobilization

22


23/37

Key Standard:

The Right Strategy?

23


24/37

The Iraq Intelligence Problem Post-war critiques have

rightly focused on thefailures of analysis withregard to WMD

Broad acceptance of thedangers of getting on boardthe assumption train -- USand British WMD whitepapers.

Limited recognition of theissues of strategywe keptdealing with Iraq as anintermittent crisis than astrategic challenge.

Potential biases in assessingthe culpability of analysisversus collection. The lasting fingerprints of

analysis Poor collection means

what?

DNI is pressing to develop

intelligence strategies forkey problems.

24


25/37

Key Standard: Lasting Policy

Consequences?

25


26/37

Indian Nuclear Test Timeline 18 May 1974 -- First Indian nuclear test

11, 13 May 1998 -- India conducts nuclear tests 13 May -- US imposes sanctions on India

28, 30 May -- Pakistan conducts nuclear tests

30 May -- US imposes sanctions on Pakistan

6 November -- US waves several economic sanctions

on both countries

26


27/37

Key Standard:

Given the requisite priority?

27


28/37

The Role of Prioritization No intelligence communityeven that of the US -- can coverthe world.

Identification of the most

critical priorities is fundamentalto the intelligence enterprise.

Lower priorities get lesscoverage.

Intelligence failure canreflect poor prioritization orappropriate prioritization thatresults in limited intelligencecapability in areas of the worldthat later become of concern.

In the US, the NationalIntelligence PrioritiesFramework is used by seniorIC leaders to guide and inform

decisions concerning theallocation of collection andanalytic resources . . . .

It reflects a dialogue withsenior policy customers onissues that matter most to

them.

28


29/37


30/37

What We Mean When We Say: An Explanation of Estimative Language

We use phrases such as we judge, we assess, and we estimateand probabilistic terms such as probably and likelyto convey analytical assessments and judgments. Such statements are not facts, proof, or knowledge. These

assessments and judgments generally are based on collected information, which often is incomplete orfragmentary. Some assessments are built on previous judgments. In all cases, assessments and judgments are notintended to imply that we have proof that shows something to be a fact or that definitively links two items or

issues.

In addition to conveying judgments rather than certainty, our estimative language also often conveys 1) our assessed

likelihood or probability of an event; and 2) the level of confidence we ascribe to the judgment.

Estimates of Likelihood. Because analytical judgments are not certain, we use probabilistic language to reflect the

Communitys estimates of the likelihood of developments or events. Terms such as probably, likely, very likely, oralmost certainly indicate a greater than even chance. The terms unlikely and remote indicate a less than evenchance that an event will occur; they do not imply that an event will not occur. Terms such as might or may reflectsituations in which we are unable to assess the likelihood, generally because relevant information is unavailable,sketchy, or fragmented. Terms such as we cannot dismiss, we cannot rule out, or we cannot discount reflect anunlikely, improbable, or remote event whose consequences are such that it warrants mentioning.

Confidence in Assessments. Our assessments and estimates are supported by information that varies in scope, quality

and sourcing. Consequently, we ascribe high, moderate, or low levels of confidence to our assessments, as follows: High confidence generally indicates that our judgments are based on high-quality information, and/or that thenature of the issue makes it possible to render a solid judgment. A high confidence judgment is not a fact or a

certainty, however, and such judgments still carry a risk of being wrong.

Moderate confidence generally means that the information is credibly sourced and plausible but not of sufficientquality or corroborated sufficiently to warrant a higher level of confidence.

Low confidence generally means that the informations credibility and/or plausibility is questionable, or that the

information is too fragmented or poorly corroborated to make solid analytic inferences, or that we have significantconcerns or problems with the sources.

30


31/37

31


32/37

Scoring the Annual Worldwide

Threat Testimony Briefing to Congress from

1996 to 2012

Scored on whether a topic

mentioned and its order ofpresentation

Provides crude metrics

Consistency suggests,

however, that there isinformation value

Raises some pertinentquestions

Strategic Threats ClearlyNoted: Afghanistan,China, Iran, Iraq, North

Korea, Proliferation,Russia, Terrorism

Some Potential Anomalies

Cyber warnings from

1996-2001, nothighlighted again until2007

Economic security rarelyhighlighted

32


33/37

Trendlines:

Transnational Threats*

0

5

10

15

20

25

30

35

40

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Terrorism

Organized Crime

Proliferation

33*Lower number equals higher priority


34/37

Reduction in

Russias Threat Profile*

0

5

10

15

20

25

30

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Terrorism

Proliferation

Russia (FSU)

34*Lower number equals higher priority


35/37

Threat Perceptions In Wartime*

0

5

10

15

20

25

30

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Balkans (Bosnia) (Kosovo)

Afghanistan (Kosovo)

Iraq (Kosovo)

35Lower number equals higher priority


36/37

What is to be done? Need for more focused questioning of the basis for a

judgment of intelligence failure (and, for that matter,

success).

Explicit engagement by intelligence oversight in helping

to establish a common framework for assessing

intelligence success and failure. More skepticism regarding claims that cultural factors

are responsible for incidences of intelligence failure

36


37/37

The Culture Critique Intelligence failures are

caused by negative culturaltraits of intelligenceorganizations.

(T)he agencies comprisingthe US intelligencecommunity are, in manyrespects, unique bureaucratic

entitities, operating fardifferently than comparablelarge American corporationsand government.

(Source: Sins of Omissionand Commission: StrategicCultural Factors and USIntelligence Failures Duringthe Cold War, Intelligenceand National Security,August 2011)

Does not explain similarfailures outside the realm ofintelligence (i.e., thefinancial crisis of 2008).

Documents

Judging Intelligence Success and Failure (Kringen)