View
236
Download
0
Category
Preview:
Citation preview
7/31/2019 Judging Intelligence Success and Failure (Kringen)
1/37
Getting Beyond
She Said and He Said
1
7/31/2019 Judging Intelligence Success and Failure (Kringen)
2/37
The Premise There has been insufficient public discussion about
the appropriate benchmarks for evaluating the
performance of the intelligence community. The consequence is that public commentary tends to
reflect efforts to spin the latest developments.
The result is a debate that is largely reactive and
lacking in perspective.
2
7/31/2019 Judging Intelligence Success and Failure (Kringen)
3/37
How Did the IntelligenceCommunity Perform?
3
7/31/2019 Judging Intelligence Success and Failure (Kringen)
4/37
Apparently perspective matters
4
7/31/2019 Judging Intelligence Success and Failure (Kringen)
5/37
Senator Feinstein
SSCI Chair Our intelligence, and I
see it all, is way behind
the times. It is
inadequate.
The IC provided her
nothing that we didnt
read in the newspapers.
Lack of human
intelligence assets and
failure to use open source.
5
7/31/2019 Judging Intelligence Success and Failure (Kringen)
6/37
Representative Rogers
HPSCI Chair Intelligence clearly helps
us understand
developments in places
like Egypt, but it is not a
crystal ball. We have got
to be realistic about its
limits, especially
regarding the complex and
interactive behavior of
millions of people.
6
7/31/2019 Judging Intelligence Success and Failure (Kringen)
7/37
First By What It is Not
7
7/31/2019 Judging Intelligence Success and Failure (Kringen)
8/37
An intelligence failure is not: A matter of leaving
intelligence consumersin the suburb of
Surprise.
The world is anuncertainplace: Hewho lives by crystal ball
soon learns to eat groundglass. Edgar R.Fieldler
8
7/31/2019 Judging Intelligence Success and Failure (Kringen)
9/37
Intelligence failure is not: A development that
results in awkwardnessor embarrassment.
Intelligence failure
implies serious long-
term consequences.
9
7/31/2019 Judging Intelligence Success and Failure (Kringen)
10/37
Intelligence failure is not: An inability to keep up with the
latest reporting.
Director of NationalIntelligence Not Briefed onLondon Arrests BeforeInterview ABC News
10
7/31/2019 Judging Intelligence Success and Failure (Kringen)
11/37
And, it probably cant do
better than . . .
The forecasts of the individuals directly involved in
driving the situation (e.g. Russia-Georgia conflict of 2008) The ability of organizations to make estimates regarding
the outcomes of multi-year planning in their own
organizations (e.g, timelines for weapons development
and procurement). The adequacy of the base information available.
11
7/31/2019 Judging Intelligence Success and Failure (Kringen)
12/37
Its All About Risk Management
12
7/31/2019 Judging Intelligence Success and Failure (Kringen)
13/37
Understanding
the Dimensions of Risk Policy risk (e.g., the UBL
operation)
Political risk (operation to
rescue US hostages in
Iran)
Human risk ( Camp
Chapman)
Operational risk (e.g,
counterintelligence)
Investment risk (e.g., UAV
captured in Iran)
Lack of Investment risk
(principally, the failure to
collect criticalintelligence)
13
7/31/2019 Judging Intelligence Success and Failure (Kringen)
14/37
Lessons Learned: 2009 Attack
on CIA Base in Afghanistan Enforce greater discipline
in communications
Apply the skills and
experience of senior
officers more effectively
Require greater
standardization of securityprocedures
More carefully manage
information sharing with
other intelligence services
Strengthen our attention to
counterintelligence
concerns
Maintain our highoperational tempo against
terrorist targets
14
7/31/2019 Judging Intelligence Success and Failure (Kringen)
15/37
Fire-fighting: Common
Preincident Intelligence Failures The first and important
failure is in identifying
occupancy types and their
associated hazards.
The second common
failure is not becoming
oriented with the facility.
A third common failure is
improper information
management.
A fourth common failure
is not participating in
exercises.
The fifth common failure
is lack of information
maintenance.
The sixth common failureis not applying the lessons
learned.
15
7/31/2019 Judging Intelligence Success and Failure (Kringen)
16/37
16
7/31/2019 Judging Intelligence Success and Failure (Kringen)
17/37
How difficult was the intelligence problem?
Was there a coherent and focused intelligence
strategy?Did the alleged failure have significant and
lasting policy consequences?
Did the intelligence community execute theintelligence priorities provided by theexecutive and legislative branches ?
17
7/31/2019 Judging Intelligence Success and Failure (Kringen)
18/37
Key Questions: Part DeuxDid the assessment appropriately
characterize the key uncertainties?
How effective was the intelligence
community in highlighting areas of
significant concern?
Did the community provide insight into
potential policy options?
18
7/31/2019 Judging Intelligence Success and Failure (Kringen)
19/37
Benchmarks from the
Business World (A)ccurate forecasting in
the economic and businessworld is usually not
possible.
The future is never exactlylike the past.
It does not seem that theseriousness of the (2008)financial crisis, or the
resulting economicrecession, was predicted bythe great majority offorecasters.
The difficulty for decisionand policy makers . . . is tounderstand and accept theextent of future uncertainty
so that appropriate plans . . .can be formulated.
Source: Forecasting anduncertainty in the economic
and business world,International Institute ofForecasters, 2009
19
7/31/2019 Judging Intelligence Success and Failure (Kringen)
20/37
Benchmarks from the
Medical Profession Although expert witnesses
greatly determine theoutcomes of medical
malpractice claims, thereliability of experttestimony is frail. (Source:Poor Agreement AmongExpert Witnesses in BialDuct Injury MalpracticeLitigation: An ExpertPanel Survey Annals ofSurgery, November 2008
20
7/31/2019 Judging Intelligence Success and Failure (Kringen)
21/37
Key Standard: Recognizing
the Degree of Difficulty Not All Intelligence
Challenges are the Same
Recall the Belgian military
intelligence officer who was
only wrong twice
21
7/31/2019 Judging Intelligence Success and Failure (Kringen)
22/37
How Hard is the
Intelligence Challenge?Less Hard More Hard
Matters of fact or history
Does Iraq Have WMD?
Where is the Federal
Directorate of Supply and
Procurement?
Is Kim Chong-il Alive?
Developments based on a longand consistent track record
Decisions that are made
transparently
The outcomes of decisions that
have not yet been made or are
opaque.
Georgian military plans in
2008
Black Swans (e.g.,
discontinuous change) The outcomes of political-
social mobilization
22
7/31/2019 Judging Intelligence Success and Failure (Kringen)
23/37
Key Standard:
The Right Strategy?
23
7/31/2019 Judging Intelligence Success and Failure (Kringen)
24/37
The Iraq Intelligence Problem Post-war critiques have
rightly focused on thefailures of analysis withregard to WMD
Broad acceptance of thedangers of getting on boardthe assumption train -- USand British WMD whitepapers.
Limited recognition of theissues of strategywe keptdealing with Iraq as anintermittent crisis than astrategic challenge.
Potential biases in assessingthe culpability of analysisversus collection. The lasting fingerprints of
analysis Poor collection means
what?
DNI is pressing to develop
intelligence strategies forkey problems.
24
7/31/2019 Judging Intelligence Success and Failure (Kringen)
25/37
Key Standard: Lasting Policy
Consequences?
25
7/31/2019 Judging Intelligence Success and Failure (Kringen)
26/37
Indian Nuclear Test Timeline 18 May 1974 -- First Indian nuclear test
11, 13 May 1998 -- India conducts nuclear tests 13 May -- US imposes sanctions on India
28, 30 May -- Pakistan conducts nuclear tests
30 May -- US imposes sanctions on Pakistan
6 November -- US waves several economic sanctions
on both countries
26
7/31/2019 Judging Intelligence Success and Failure (Kringen)
27/37
Key Standard:
Given the requisite priority?
27
7/31/2019 Judging Intelligence Success and Failure (Kringen)
28/37
The Role of Prioritization No intelligence communityeven that of the US -- can coverthe world.
Identification of the most
critical priorities is fundamentalto the intelligence enterprise.
Lower priorities get lesscoverage.
Intelligence failure canreflect poor prioritization orappropriate prioritization thatresults in limited intelligencecapability in areas of the worldthat later become of concern.
In the US, the NationalIntelligence PrioritiesFramework is used by seniorIC leaders to guide and inform
decisions concerning theallocation of collection andanalytic resources . . . .
It reflects a dialogue withsenior policy customers onissues that matter most to
them.
28
7/31/2019 Judging Intelligence Success and Failure (Kringen)
29/37
7/31/2019 Judging Intelligence Success and Failure (Kringen)
30/37
What We Mean When We Say: An Explanation of Estimative Language
We use phrases such as we judge, we assess, and we estimateand probabilistic terms such as probably and likelyto convey analytical assessments and judgments. Such statements are not facts, proof, or knowledge. These
assessments and judgments generally are based on collected information, which often is incomplete orfragmentary. Some assessments are built on previous judgments. In all cases, assessments and judgments are notintended to imply that we have proof that shows something to be a fact or that definitively links two items or
issues.
In addition to conveying judgments rather than certainty, our estimative language also often conveys 1) our assessed
likelihood or probability of an event; and 2) the level of confidence we ascribe to the judgment.
Estimates of Likelihood. Because analytical judgments are not certain, we use probabilistic language to reflect the
Communitys estimates of the likelihood of developments or events. Terms such as probably, likely, very likely, oralmost certainly indicate a greater than even chance. The terms unlikely and remote indicate a less than evenchance that an event will occur; they do not imply that an event will not occur. Terms such as might or may reflectsituations in which we are unable to assess the likelihood, generally because relevant information is unavailable,sketchy, or fragmented. Terms such as we cannot dismiss, we cannot rule out, or we cannot discount reflect anunlikely, improbable, or remote event whose consequences are such that it warrants mentioning.
Confidence in Assessments. Our assessments and estimates are supported by information that varies in scope, quality
and sourcing. Consequently, we ascribe high, moderate, or low levels of confidence to our assessments, as follows: High confidence generally indicates that our judgments are based on high-quality information, and/or that thenature of the issue makes it possible to render a solid judgment. A high confidence judgment is not a fact or a
certainty, however, and such judgments still carry a risk of being wrong.
Moderate confidence generally means that the information is credibly sourced and plausible but not of sufficientquality or corroborated sufficiently to warrant a higher level of confidence.
Low confidence generally means that the informations credibility and/or plausibility is questionable, or that the
information is too fragmented or poorly corroborated to make solid analytic inferences, or that we have significantconcerns or problems with the sources.
30
7/31/2019 Judging Intelligence Success and Failure (Kringen)
31/37
31
7/31/2019 Judging Intelligence Success and Failure (Kringen)
32/37
Scoring the Annual Worldwide
Threat Testimony Briefing to Congress from
1996 to 2012
Scored on whether a topic
mentioned and its order ofpresentation
Provides crude metrics
Consistency suggests,
however, that there isinformation value
Raises some pertinentquestions
Strategic Threats ClearlyNoted: Afghanistan,China, Iran, Iraq, North
Korea, Proliferation,Russia, Terrorism
Some Potential Anomalies
Cyber warnings from
1996-2001, nothighlighted again until2007
Economic security rarelyhighlighted
32
7/31/2019 Judging Intelligence Success and Failure (Kringen)
33/37
Trendlines:
Transnational Threats*
0
5
10
15
20
25
30
35
40
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Terrorism
Organized Crime
Proliferation
33*Lower number equals higher priority
7/31/2019 Judging Intelligence Success and Failure (Kringen)
34/37
Reduction in
Russias Threat Profile*
0
5
10
15
20
25
30
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Terrorism
Proliferation
Russia (FSU)
34*Lower number equals higher priority
7/31/2019 Judging Intelligence Success and Failure (Kringen)
35/37
Threat Perceptions In Wartime*
0
5
10
15
20
25
30
1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
Balkans (Bosnia) (Kosovo)
Afghanistan (Kosovo)
Iraq (Kosovo)
35Lower number equals higher priority
7/31/2019 Judging Intelligence Success and Failure (Kringen)
36/37
What is to be done? Need for more focused questioning of the basis for a
judgment of intelligence failure (and, for that matter,
success).
Explicit engagement by intelligence oversight in helping
to establish a common framework for assessing
intelligence success and failure. More skepticism regarding claims that cultural factors
are responsible for incidences of intelligence failure
36
7/31/2019 Judging Intelligence Success and Failure (Kringen)
37/37
The Culture Critique Intelligence failures are
caused by negative culturaltraits of intelligenceorganizations.
(T)he agencies comprisingthe US intelligencecommunity are, in manyrespects, unique bureaucratic
entitities, operating fardifferently than comparablelarge American corporationsand government.
(Source: Sins of Omissionand Commission: StrategicCultural Factors and USIntelligence Failures Duringthe Cold War, Intelligenceand National Security,August 2011)
Does not explain similarfailures outside the realm ofintelligence (i.e., thefinancial crisis of 2008).
Recommended