Search Support in Information Analysis and Synthesis

Emily S. Patterson, PhDResearch Scientist

Associate Director, Converging Perspectives on Data (CPoD)

Not for distribution – please contact patterson.150@osu.edu

• Ohio State University (OSU) consortium:

• Innovate solutions to data overload (beyond “tweaks”)

• Advance methods for envisioning useful support for

information analysis and comprehension (IA&C)

• Develop interdisciplinary talent (Cognitive Systems

Engineering, Design)

• Innovate by pursuing leverage points:

1. Process vulnerability with high consequences for failure

2. “New” technological capability

3. Grounded basis for predicting performance

improvement

Converging Perspectives on Data (CPoD)

©2000 Christoffersen, Woods, Malin

Design Seed

• Modular design concept

– Generalizes across software, architecture, scenario,

domain

• Instantiated in a case

– Ideally animated mockup (ani-mock)

• Addresses process vulnerability

• Leverages technological advance

– Multiple levels for relying on machine processing

• Elicits feedback on concept “usefulness”

• Study: 10 NASIC analysts on Ariane 501• Repeated inaccurate information

“High profit” attribute-based document search Social bookmarking of documents/messages Automated event detection Circular reporting alerts (magnified phrases) Overview visualizations

• Inappropriately applied default assumption Broaden consideration of perspectives on data

• Missed update that overturned assessment Mixed-initiative update detection

Presentation Outline

Study: Observe 10 NASIC analysts (lab)

Time: 2 hour session (avg = 55 min)

Task: Causes, impacts of Ariane 501 accident

Participants: 10 NASIC analysts (avg = 13 yrs)

Tools: Search/browse features of Pathfinder, Word

Data: “On topic” database (~2000 documents)

Briefing: Verbal (video-taped)

Procotol: Think aloud, semi-structured interviews

Analysis: Process tracing, briefing accuracy

Quick Reaction Task

In 1996, the European Space Agency lost a satellite during the first qualification launch of a new rocket design. Give a short briefing about the basic facts of the incident: when it was, why it occurred, and what the immediate impacts were?

1. Repeated inaccurate information

2. Relied on default assumption

3. Missed update that changed assessment

Reasons for Inaccurate Statements

Software failureJune 4, 1996

When Why - operational contributors

Inertialreferencesystem

Backup andprimary IRS

Embeddedsoftware

No guidance databecause IRS shutdown

Diagnostic datainterpreted asguidance data

Booster andmain enginenozzlesswiveledabnormally

Rocket self-destructed

Rocketveered offcourse

Numerical overflowoccurred because thehorizontal velocityhad more digits thanprogrammed

Flight profiledifferent on A5because a fasterrocket than A4

IRS shut downbecause ofnumerical overflow

Re-usedsoftware fromAriane 4

Software notneeded afterliftoff

No protectionfor common-mode failure

Insufficienttestingrequirements

No protection fornumericaloverflow onhorizontal velocity

No integratedtesting “in the loop”

Poorcommunicationacrossorganizations

No softwarequalificationreview

Multiplecontractorspoorlycoordinated

Reviewprocess wasinadequate

What happened

Why - design and testing contributors

Why - organizational contributors

Where

Less than aminuteafter liftoff

©1999 Patterson

Inaccurate Information on Accident Contributors

Constructed Database

~ 2000 text documents• on target 60%• context 35%• off topic 5%• high profit 0.5% (9 documents)

• Highly biased or deceptive sources • Lack of expertise in the subject area• Distanced from the original data • Language translation• Predictions• “Moving target” of data, events

Sources of Inaccurate Information

File Edit Date Title

Ariane 5 Flight 501 Failure: Report by the Inquiry Board (July 19, 1996)Inertial Reference Software Error Blamed for Ariane 5 Failure; Defense Daily (July 24, 1996)Software Design Flaw Destroyed Ariane 5; next flight in 1997; Aerospace Daily (July 24, 1996)Ariane 5 Rocket Faces More Delay; The Financial Times Limited

(July 24, 1996) Flying Blind: Inadequate Testing led to the Software Breakdown that Doomed Ariane 5; The Financial Times Limited (July 25, 1996)Board Faults Ariane 5 Software; Aviation Week and Space Technology (July 29, 1996)Ariane 5 Explosion Caused by Faulty Software; Satellite News (August 5, 1996) Ariane 5 Report Details Software Design Errors ; Aviation Week and Space Technology (September 9, 1996)Ariane 5 Loss Avoidable with Complete Testing; Aviation Week and Space Technology (September 16, 1996)

“High Profit” Documents

Information Sampling by Participant 5

419 Query 2

24 on-topic

8 cut and

paste

2000 in Database

3 key

725 Query 1

6 High

Profit

3 High

Profit

28 read

Participant 5: 96 minutesExperience: 17 yearsQuery 1: ESA | (european & space & agency)Query 2: (ESA | (european & space & agency)) > (19960601) Infodate

419

28

Key documents

Key documents that are high profit

High profit documents

Legend

S2: 73 minutesesa & ariane*(esa & ariane*) & failure

S3: 24 minuteseurope 1996(europe 1996) & (launch failure)(europe 1996) & ((launch failure):%2)

S4: 68 minutes(european space agency):%3 & ariane & failure & (launcher |rocket))

S5: 96 minutesESA | (european & space & agency)(ESA | (european & space & agency)) > (19960601) Infodate

S6: 32 minutes1996 & Ariane(1996 & Ariane) & (destr* | explo*)(1996 & Ariane) & (destr* | explo*) & (fail*)

S7: 73 minutessoftware & guidance

S8: 27 minutesesa & arianeariane & 5(ariane & 5):%2((ariane & 5):%2) & (launch & failure)

S9: 44 minutes1996 & European Space Agency & satellite1996 & European Space Agency & lost1996 & European Space Agency & lost & rocket

161

29

22

5

169

15

419

28

7

18466

14 12

194

4

29

Key documents Key documents that are high profitHigh profit documents

Participants whose key documents were not high profit documentsParticipant Experience Time Final query Documents High profit docs

(years) (min.) (# hits) (# read) (# read)3 4 24 22 5 06 8 32 184 7 28 11 27 194 12 09 18 44 29 4 0

Average: 10.3 32* 107 7* 0.5*

Participants whose key documents were high profit documentsParticipant Experience Time Final query Documents High profit docs

(years) (min.) (# hits) (# read) (# read)2 8 73 161 29 34 8 68 169 15 25 17 96 419 28 27 9 73 66 14 5

Average: 10.5 78* 204 22* 3*

* Significant differences using Wilcoxon-Mann-Whitney Non-Parametric Test

More Time to Find High Profit Documents

Participants whose key documents were not high profit documents Participant Accurate Vague Inaccurate Nothing

3 5 2 2 11

6 11 1 3 5

8 9 0 0 11

9 5 3 1 11

Average: 7.5 1.5 1.5* 9.5

Participants whose key documents were high profit documents

Participant Accurate Vague Inaccurate Nothing

2 5 2 0 13

4 11 2 0 7

5 12 3 0 5

7 8 1 0 11

Average: 11 2 0* 6.75

* Significant differences using Wilcoxon-Mann-Whitney Non-Parametric Test

More Accurate with High Profit Documents

July 5, 1996 (Report 858):Ariane 5 lifts off much faster… information…exhausted the temporary memory (buffer)capacity…both systems simultaneously declaredthemselves to be in an irredemiable errorsituation and commenced a resetprocedure…when the system was reset, thevehicle’s position at that time…was adopted asthe reference base

September 16, 1996 (Report 1385):the active inertial reference system transmittedessentially diagnostic information to thelauncher’s main computer, where it wasinterpreted as flight data and used for flightcontrol calculations

Participant’s ResponseArticle Date/Content

Participant 7 Briefing: “numerical values beyond the programmed limits of the flightcomputer…the platforms initiated a diagnostic “reset” mode that fed incorrect values to theflight computer”

nothing

July 29, 1996 (Report 1440):as a result of the double failure, the active IRSonly transmitted diagnostic information tothe booster’s on-board computer, which wasinterpreted as flight data and used for flightcontrol calculations

“We know there was a problem because the guidanceplatforms shut down. After they shut down, the inertialreference system sent diagnostic information so they’redesigned to shut down when something goes wrong.Assuming the other system has taken over, it’s sendingdiagnostic information so that the people on the groundcan figure out what went wrong with it. Having themboth shut down, the guidance computer is interpretingthe diagnostic information as where it’s at and insteadof getting numbers, it’s getting other things…”

“...In this article, it says when it shut down, it started areset procedure. In the other article, it says diagnostic

information. This article and the otherone…are incompatible, inconsistent with eachother…Of course messages that can’t both be right happenall the time. I’m finding it hard to believe that the vehicleis going to fly without any inertial inputs whatsoever…let’s look at the source…FBIS report. Translatedtext…the other one was later also…it sounds good. If I

had to guess, I would go with the other one.

Think Aloud: Assessing Information Accuracy

lack of technicalexpertise

errors in translation

biased interpretations

lack of privilegedaccess

lack updates

predictions offuture events

contestable interpretation

Observations

Hypotheses

discrepant

Sourcereputation forcredibility

reputation ofbias

reputation forexpertise in aparticular area

officiality ofresponsibility to dothe analysis

Documenttemporalrelationshipto events

amount quoteddirectly fromofficial document

translateddepth andbreadth ofthemecoverage

length

Description

corroborativeInteractionsbetweendescriptions

corroboration

same sourcedeception

temporal relationshipto updates

level ofsensationalism

level of technicalsophistication

abstract

©1999 Patterson

Accuracy “Cues” and Judgments

Generic “High profit” Attribute-based Search

Feedback from Analysts:• It would be important to filter out a group like press releases, not just select them• With new searches, you change people, places, things, but not your area (communications, command and control)• I want profile to generate documents; then I want to visualize what portions contribute; then I want to see how this changes the results• I want to see what the high value input is from each query and see progress over queries

Tailored “High profit” Attribute-based Search:

Technology Forecasting

Social Bookmarking: Del.icio.us (now Yahoo)

Del.icio.us: Katrina tag

Social Tagging: Barriers to Implementation

Feedback from Analysts:• What scale is needed for emergent properties to emerge? If you’re really lucky, you might get 10,000 people to use this inside analysis agencies. If you have 50, does it work? • We want to automate tags. We want one database to talk to another. These are the big issues.• It’s only going to work to find related people if people use it. Juniors need seniors to do it to find them in the network. • Not always English, text, or open source• Distinguish between crime scene analogy vs text as data; in Iraq non-verbal stuff can tell you if someone is a friend or foe• We are doing all of this already. It’s hard to see what’s new about this. • All documents already have tags. • This requires someone to type in something. • Tags are such a big hassle that we just put down anything.

Ariane 501 Launch Failure(June 4, 1996)

Inquiry Board Report(July 19, 1996)

Ariane 502 Launch(October 30, 1997)

Early reports:details, eyewitnesses,immediate reactions,inaccuracies Summaries of

board findings

Comprehensive,long-term reactions,less diversity

Short summaries,Updates on themes

©1999 Patterson

3 Important Events in Ariane 501

ariane

EntireDatabase

failure

ariane AND failure; narrowed by time

ariane AND failure AND software; narrowed by time

501Report

502

Automated Event Detection: Query Results

“OK, this is the same one [the Ariane 501 launch]. This is after the fact. Uh oh. Remember I said how data changes? I’m looking…apparently it says a mechanical failure and then I come along. What’s this say? Failure was due to the brain. It turned out that computer software which was designed for 4, which is much slower. So it turns out now my analysis has changed. It now looks like it was an integral failure. Period. Brought on by internal software. So I’ll qualify this (draws an arrow from previous note below a line and writes “#1435 wrong software used, software for AR4 used in AR5 launch”). That was the problem. Lost guidance. Launch software.”

“Software failure. It’s a confirmation of the previous message saying it’s a software failure… or is this the same message? Yeah, yeah, that’s where the highlighter…”

November 3, 1997 (second time):

a software failure caused the rocket to veer off course and

fall apart August 4, 1996:

failure was due to Ariane-5’s “brain.” It turned out that

the computer software in the Ariane-5 was originally

designed for the Ariane-4, a much slower rocket. Seconds

after take-off, Ariane-5 reached a velocity that exceeded the “brain’s”

computing capacity. It lost all guidance and attitude

information, and the on-board computer tried to supercede the software programme and activated the rocket’s solid

fuel propellant boosters

Participant’s ResponseArticle Date/Content

nothing

November 3, 1997 (third time):

a software failure caused the rocket to veer off course and

fall apart

Participant 9: Illusory Confirmation

Circular Reporting Protection: Magnified Phrases

Feedback from Analysts:• I want to copy pieces, not just whole document• I don’t want to have to always highlight phrases• I want multiple phrases shown from where in the document• Add marking function to come back later (reminder)• Add freetext “stickies” • Add manual tags:

• reliable, not reliable, questionable, no evidence

© 2005 Patterson

Extension: Tagging Magnified Phrases

Feedback from Analysts:• All: Want both folders and tags; add hierarchies (1000 tags; 1600 folders (range 450-2100))• All: Want for >1 project; want to flag/file under other topics than working on• Want to see emerging themes; need support for reconfiguring as model changes• Pre-generate tags without typing, but also give flexibility (no strict ontology)• Automated tags (classification) on bottom and my tags on top • I put like things together to help with focus and understanding, but not everyone does• 425 tags are too many• Tags are such a hassle that we put anything• I print and stack by topics and studies so I’m not sure if I’d use this• Want to sort/view differently: by classified/unclassified, humint/sigint, date/source/title/classification

© 2005 Patterson

“Low-profit” key articles “High-profit” key articlesEurope: Causes of Ariane 5Failure(July 5, 1996)

Software design flawdestroyed Ariane V; nextflight in 1997(July 24, 1996)

Ariane 5 Failure: InquiryBoard Findings(July 25, 1996)

Board Faults Ariane 5Software(July 29, 1996)

False computer commandblamed in Ariane V failure(June 6, 1996)

Ariane 5 loss avoidable withcomplete testing(September 16, 1996)

Low and High Profit Documents

report dates

report space

disruptingevent

event thread

prediction of future event

analysis ofpast event

Visual Narratives

(ongoing plan)

Selection Mechanism (e.g., keyword search)

Epoch

landmark event

updates

Models of document types (e.g., high profit document)

Models of scenario elements(e.g., accident)

© 1999 Tinapple, Woods, Patterson

Overview Visualization: Documentspace + Themespace

Overview + Detailed Integrated Workspace

1. Repeated inaccurate information

2. Relied on default assumption

3. Missed update that changed assessment

Reasons for Inaccurate Statements

Participant 3: Inaccurate Statement

…the basic impact of the launch is a failure. The monetary loss can be recovered by the insurance...

Participant 6: Confidence Assessment

I am very confident. Everybody agrees. It was the official inquiry board. The reports, they weren’t written by the French, they were written by other people and they don’t disagree. If they did, they would say so.

Broaden Assessment of Perspectives on Data

• Provide “perspective” information on demand:• Organizational role

• Spatial

• Temporal

• Political group affiliation

• Query expansion suggestions • SS-4 missile = R-12 missile

• Cuban missile crisis = Caribbean crisis; October crisis

• Similar query detection

• Find who has opened similar documents

• Find who has written similar analyses

1. Repeated inaccurate information

2. Relied on default assumption

3. Missed update that changed assessment

Reasons for Inaccurate Statements

Ariane 5 Program Impacts

Loss of rocket booster

Loss of payload

Delay A5 qualification

Delay 502 launch

No paying customer for 503

Cluster Satellite Program Impacts

Loss in market share

Ariane 4 Program Impacts

Program extended

Additional launchers ordered

Insurance rates rise

No 502payload

Delay 503 launch

Program cancelled

Rebuild 1

Additional funds found: rebuild 4

Cannot launch on A5: launch on Soyuz

Inaccurate Information on Accident Impacts

© 1999 Patterson

(Inaccurate?) Predicted Launch Delays

Prior to 501 Launch September ‘96

Right after 501 incident(June 4, ‘96) December ‘96

After Inquiry Board Report (July 19, ‘96)

December ‘96

March ‘97

Actual 502Launch Date

March - June ‘97

July ‘97

September ‘97

October 30, ‘97

Projected Launch Date

Announcement DateJune 96

July

Aug

Sept

Oct

Nov

Dec

Jan 97

Feb

Mar

Apr

May

June

July

Aug

Sept

Oct

Nov

June 96

July

Aug

Sept

Oct

Nov

Dec

Jan 97

Feb

Mar

Apr

May

June

July

Aug

Sept

Oct

Nov

Launch Date

Report Date

Ariane 4 Program

(Inaccurate?) Predicted Disruptions to Plans

502

Ariane 5 Program

ClusterSatellite Program

lost satellites(no insurance)

programcancelled rebuild 1

additionalfunds found:rebuild 4

cannot launch on Ariane 5: launch on Soyuz

investigationdelay 502 launch

no 502payload

delay 503launch

no 503 payload

programextended

additional launchers ordered

insurance rates rise

501

Study Limitations

• Small number of study participants

• Single scenario

– Correct answer known

– Prediction plays a minor role

– No prior experience in knowledge area

– Does not include intentional deception

• Simulated rather than naturalistic setting

– No ability to print documents

– No ability to access other analysts

– Unfamiliar tool and restricted feature use (Pathfinder)

• Verbal protocols more “read aloud” than “think aloud”