Institute for Collaborative Innovation (ICI) 2006: Rigor in...

Institute for Collaborative Innovation (ICI) 2006: Rigor in Information Analysis

Emily S. Patterson, PhD

Research Scientist Associate Director, CPoD

Ohio State University Proprietary – please contact patterson.150@osu.edu !

Overview of research/design on data overload

Rigor

  Study

  Strategies

  Process Metric

Angola 2011 Scenario-based exploration of design ideas

Outline

Ohio State University (OSU) consortium:

  Innovate solutions to data overload (beyond “tweaks”)

  Advance methods for envisioning useful support for information

analysis and comprehension (IA&C)

  Develop interdisciplinary talent (Cognitive Systems Engineering,

Design)

Innovate by pursuing leverage points:

  Process vulnerability with high consequences for failure

  Innovative technological or organizational capability

  Grounded basis for predicting performance improvement

Converging Perspectives on Data (CPoD)

Cross-Cutting Research Themes

1.  Integrated end-to-end workflow

2.  Break hypothesis fixation using multiple perspectives

3.  Find critical data via emergent collaboration

4.  “Actionable” intelligence by judging analytic rigor

Down Collect Conflict and Corroboration

Hypothesis Exploration

CPoD Intelligence Analysis Studies

1: Interview 5 analysts on workflow differences

2: Perspectives case studies: Libyan shootdown, Zaire, Middle East

3: Observe 4 teams collaborating on WWII counter-insurgency

4: Interview 10 analysts on process rigor behind two products

Observe 10 experts on Ariane 501 rocket accident Interview 46 analysts, instructors, support personnel Observe team during Stability and Support Ops (SASO) Interview 6 experts who critique novice on Ariane 501

Foundational Research: Challenges in Intelligence Analysis

Targeted Inquiry on Research Themes

©2000 Christoffersen, Woods, Malin

Jumpstarting Innovation with Research

Design Seed

Modular design concept

  Generalizes across software, architecture, scenario, domain

Instantiated in a case

  Animated mockup (ani-mock)

Addresses process vulnerability

Leverages technological advance

  Multiple levels for relying on machine processing

Elicits feedback on concept “usefulness” and state of technology

CPoD “Seed Book” for Intelligence Analysis

“High profit” attribute search Magnified phrases workspace Circular reporting alerts Automated “bookkeeping” functions Automated event detection Mixed-initiative update detection Visual narratives workspace

Theme 1: Integrated workflow

Query expansion based on analyst perspective (bias) Automated detection of similar queries & analyses

Theme 2: Multiple perspectives

“Open” spaces for cross-checking Meta-tag ontologies for “snippets”

Theme 3: Emergent collaboration

Institute for Collaborative Innovation (ICI) 2006

Theme 4: Improve and assess rigor

Definition: Rigor

for

of

Strategies to Increase Rigor (apprenticeship learning from gurus)

Peer review Specialist consults Devil’s advocates

Personal “vetted” files “Read-on” immersion

Read for and against Prediction diversity sampling Perspective (bias) analysis

Read old to new Timelines

“Build a house” around question 360 degree analysis

Corroborate data Separate facts and assumptions Reminders of uncertainty Track “to be included” info

Organize documents Notesheet organization

Referencing

NASA Columbia Accident Investigation

"Another lack of rigor cited by the panel is the widespread use of PowerPoint presentations in lieu of actual engineering data and analyses.”

Strategies to Infer Rigor

Relationship to others: - Consensus view - Extreme opinion

Reasoning: - Warrant for inferences - Believability - Certainty

Analyst competence: - Reputation (individual, organization) - Ability to answer questions - Credentials - Professionalism Presentation:

- Clarity - Polish

Down Collect Conflict and Corroboration

Hypothesis Exploration

Broadening checks (“up” arrows) increase rigor

Elm, W., Potter, S., Tittle, J., Woods, D.D., Grossman, J., Patterson, E.S. (2005). Finding decision support requirements for effective intelligence analysis tools. Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting.

Aspects of (Sufficient) Process Rigor

Analyst - Analyst

Analyst - Supervisor

Supervisor - Policymaker

Comparison of Two Processes

The End

©1999 Tinapple, Woods, and Patterson

Exploration of Concept Usefulness

Example: Al Qaeda Iraq Pre-War Connection

Principle Component 1

Principle Component 2

Little or no connection

Neutrals Strong connection

High profits

Compressed semantic space

“911 Report”

Mylroie testimony

Query

“The Real Connection”

Example: Search Results

Query: Al Qaeda Iraq Connection

*Rigor by sampling best of multiple perspectives

Nearest Neighbor (google ‘98)

Diversity (CMU)

Tuned Diversity* (Our vision)

1 Selling An Iraq Al Qaeda Connection

1 Selling An Iraq Al Qaeda Connection

911 Report

2 General…Connection …is Nonexistent

2 Bush Defends Assertions…

Mylroie Testimony

3 Ties…Footnote Fahrenheit 3 Poverty and Low Education…

The Real Connection

Example: Hypotheses and Sources

Connection Hypotheses Key Sources

Strong Cell leader worked for Saddam 911 Report

& Mylroie testimony

Medium Cell leader met with Saddam 911 Report

& Mylroie testimony

Saddam developed insurgent Al Qaeda contacts

Weak Saddam made a mural celebrating 911 The Real Connection

Masri and Zarqawi started a Bagdad cell in 2002

None There was no connection. The Real Connection

Confirmed Disconfirmed

Future Directions

In down-collect, analyst wants: Awareness of multiple perspectives → diversity goal High profit documents → tuning goal

Planned Contributions: “meta-methods” to tune index and search parameters

Insights (e.g., which settings foster speed and profit) NOT replace existing indexing methods and search engines

Run Analyst … Engine Parm. 4 # Errors Speed (ms)

1 Fred … 0.92 4 62

2 Sally … 0.20 0 33

40 Sally … 0.20 3 115

Example experimental plan

The End

Innovations for Information Analysis and Comprehension A Consortium at The Ohio State University

Distributed Tasking

Institute for Collaborative

Innovation

for Hypothesis Generation

Kidnapping is for ransom from oil

company

(electronics are expenditure)

Xt = Xt-1

espionage

on-line propaganda

insurgent-style video terrorism

cyber-attacks

selling stolen equipment for $

The End

Likelihood

Ris

k

H0

H1

H2

H3

Data 1

Data 27

Data 33

Data 45

Likelihood

Ris

k

H0

Data 1

Data 27

Data 33

Data 45

Likelihood

Ris

k

H0

H1

H0.1

H2

H3

Rebel/Congo joint attack planned

Likelihood

Ris

k

Rebels have gained outside support

Rebels planning attack

Chinese language intercepted on Govt. channels

Rebel attacks increase Rebels have gained support from Congo

Rebels have gained support from Mid East

Day 1 Day 2

Evidence of rebel planning activity

Evidence of rebel planning activity

Evidence of rebel planning activity

No French or Kituba language traffic

Rebels planning attack Rebel/Mid

east terrorist joint attack planned

Arabic language traffic on rebel channels

Day 3

Mid East business deal is benign

Chinese planning to invest in Angolan oil

Rebels/Mid East allied to destroy embassy/ derail Chinese oil investment.

Chinese embassy implicated as rebel target

Chinese message translated: “… oil facility security…”

Arabic signals are regarding a benign business deal

Mid east bomb-making materials being smuggled

Anti-rebel actions have had an effect

The End

BRIEFING INTERACTION

BRIEFING INTERACTION

MESSAGE BRIEFER

AUDIENCE LIMITED FEEDBACK

Briefing Interactions: Traditional Model

BRIEFING INTERACTION CYCLE

BRIEFING ���ANALYST AUDIENCE���

PARTICIPANTS

ANALYSIS BRIEFING

INFORMING

TASKING

Analyst���Supervisor

Decision Maker

Briefing Interactions: Participatory Exchange Model

BRIEFING INTERACTION CYCLE

BRIEFING ���ANALYST AUDIENCE���

PARTICIPANTS

ANALYSIS BRIEFING

INFORMING

TASKING

ANALYSIS REPLANNING

REPLANNING ANALYSIS

CONVERGING PERSPECTIVE

Analyst���Supervisor

Decision Maker

Briefing Interactions: Participatory Exchange Model

The Participatory Exchange

•  Driven by Participatory Interaction

• Conversation with the Audience

• Supports Re-Tasking

• Approach is Supported by...

- Empirical Analyst Reports

- LNG Scenario Walkthrough Study

- Laws of Cognitive Work

The Participatory Exchange: Summary

Active Participant Familiarity with discussion topic Understanding of unique viewpoint

Guides through data space Integrates feedback Unifies idea themes Controls flow of the exchange

Presenter Audience

The End