CLE - February 2009 - Jonas - final pdf€¦ · Enterprise Intelligence vs. Enterprise Amnesia Jeff Jonas Chief Scientist, IBM Entity Analytics IBM Distinguished Engineer

Enterprise Intelligencevs. Enterprise Amnesia

Jeff JonasChief Scientist, IBM Entity AnalyticsIBM Distinguished Engineer

© 2009 IBM Corporation© 2009 IBM Corporation

The data is the question.

More data is better and faster.

Bad data is good.

Intelligence: What If?


1983: Founded Systems Research & Development (SRD)

1992: Assisted Vegas casinos in detecting the unwanted – resulting in a technology known as NORA

1996: Developed a consolidated consumer database drawing from 4,200 disparate systems, modeling 80M consumers and their 1.6B transactions

2001/2003: Funded by In-Q-Tel, the CIA’s venture capital arm

2003: Hired a CEO

2005: Acquired by IBM, now Chief Scientist, IBM Entity Analytics

Today: Focus is in the area of ‘sensemaking on streams’ with special attention towards privacy and civil liberties protections

Background


”The data must find the data … and the relevance

must find the user!”


State of the Union:Enterprise Amnesia


EmployeeDatabase

InvestigationsDatabase

Human Resources

CorporateSecurity

Enterprise Amnesia

Sales and Service

CustomerDatabase


EmployeeDatabase


The newly hired employee had previously been arrested for stealing from you … same store!

Human Resources

CorporateSecurity

Enterprise Amnesia

Sales and Service

CustomerDatabase


Amnesia is Embarrassing

Amnesia is Expensive


Time

Com

puti

ng P

ower

Gro

wth

All Digital Data

SensemakingAlgorithms

Amnesia Index

Faster Computers Making Us Dumber


is the leading cause of

enterprise amnesia.

Perception isolation


Enterprise Intelligence

requires

“Persistent Context.”

The Brain!


[email protected]

Information Without Context


[email protected]

Information Without Context


[email protected]

Information Without ContextData


Information In Context


[email protected]



NewsletterSubscriber

Top 200Customer

Term“no rehire”

Pending Investigation

[email protected]



Introducing Persistent Context


“Remembering in a database (persistent) how things relate to each other (context)”

Introducing Persistent Context


Context Accumulation: Think Pixels to Pictures

An assertion (connects, near its neighbors, or un-associated) is made with the arrival of each new piece

Assertions favor the false negative

New connecting pieces … create new entities that are re-evaluated against the remaining entities

Some pieces produce epiphanies

New pieces sometimes reverse earlier assertions

Working space quickly begins to exceed the final size

There can come a tipping point – a collapsing of the working space


Observations

Uni

que

Iden

titi

es

Actual Identities

Six (6) Liars Detected Here

More, Better, Smarter & Faster


Ingesting 800Menterprise

observations in 4 days!

Very Fast

P595– 32 CPU (64 SMT)– 512GB RAM– 32x 1GB adaptors

4x DS4800 controllers– 14x EXP810 trays with 36GB 15k RPM drives

1x DS4700 controllers

6x BladeCenters– 14x HS21 blades (4 cores each)

1x Force 10 switch


ProspectDatabase

Human Resources

EmployeeDatabase

CorporateSecurity


Sales and Service

CustomerDatabase

The “Data is the Query” Beats “Boil the Ocean”


ProspectDatabase

Human Resources

EmployeeDatabase

CorporateSecurity


Sales and Service

CustomerDatabase

Batch Analytics

The “Data is the Query” Beats “Boil the Ocean”


Persistent Context

ContextAnalysis

Relevance Detection

FeatureExtract & Enhance

Publish

Notice Respond

Sensemaking on Streams … How To


Persistent Context

ContextAnalysis

RelevanDetecti

FeatureExtract & Enhance

Notice

ObservationsStructured

UnstructuredAudio/VideoGeospatialBiometrics

Etc.

QuestionsSearch, Discovery,Context Requests

Etc.



sistent Context

extysis

Relevance Detection

Publish

Respond

CONSUMERSOperational SystemsBusiness IntelligenceData MiningPattern DiscoveryPredictive ModelingCase ManagementVisualizationEtc.

Answers to questions



1st Principle

If you do not process new enterprise observations first like a question … then you will

not know if it matters …unless someone asks.


What Makes a Smart System … Smart?

Dynamic and resilient “domain in-specific”engineering

No such thing as a single version of truth

Bad data good

New observations reverse earlier assertions


Drivers?

Competition.


Human Capital

ToolsData

Domains for Competitive Advantage


Human Capital

ToolsData

FastestSensemaking



Human Capital

ToolsData

First

FastestSensemaking



“Every millisecond gained in our program trading applications is worth $100 million a year.”

Goldman Sachs, 2007 * Source Automated Trader Magazine 2007


The Way Forward


Time

Com

puti

ng P

ower

Gro

wth

All Digital Data


Amnesia Index

Faster Computers Making Us Dumber


Time

Com

puti

ng P

ower

Gro

wth


All Digital Data

The Way Forward


Time

Com

puti

ng P

ower

Gro

wth

New/Useful Information


All Digital Data

Data Reduction

The Way Forward


Time

Com

puti

ng P

ower

Gro

wth

New/Useful Information


All Digital Data

Data Reduction

ContextEngines

The Way Forward


Enterprise Service Bus

ContextEngine

TransactionalActivity

Prospects

CustomerAcquisition

Call CenterCalls

Employees and Applicants

Vendors

Marketing Campaigns

Investigations and Arrests

Context Engine: A General Service


1. Opportunity/risk analysis at point of transaction

2. Know your customer beyond householding – leverage social circles

3. Improve data quality - at collection

4. Higher dimensional data for improved modeling, statistics and management

5. Perpetual analytics for real-time changes to risk/opportunity

Enterprise Intelligence - For Example:


y

x

good

bad

Risk ……… Moves


y

x

good

bad



y

x

good

bad



y

x

good

bad

Mortgage Backed Securities?


Smart Planet

New Intelligence

Information Agenda

Trusted Information


FinancialServices

Retail

HealthCare

Energy

CounterTerrorism

CyberSecurity

SocialServices

PharmaResearch

Context Engines: Whole-Spectrum Benefits


Blog References

Puzzling: How Observations Are Accumulated Into Context

Smart Systems Flip-Flop

When Risk Assessment is the Risk

Context: A Must-Have and Thoughts on Getting Some …

Algorithms At Dead-End: Cannot Squeeze Knowledge Out Of A Pixel

You Won't Have to Ask - The Data Will Find Data and Relevance Will Find the User

Streaming Analytics vs. Perpetual Analytics (Advantages of Windowless Thinking)

Federated Discovery vs. Persistent Context – Enterprise Intelligence Requires the Later

It Turns Out Both Bad Data and a Teaspoon of Dirt May Be Good For You

How to Use a Glue Gun to Catch a Liar


Enterprise Intelligencevs. Enterprise Amnesia

Jeff JonasChief Scientist, IBM Entity AnalyticsIBM Distinguished Engineer

Documents

CLE - February 2009 - Jonas - final pdf€¦ · Enterprise Intelligence vs. Enterprise Amnesia Jeff Jonas Chief Scientist, IBM Entity Analytics IBM Distinguished Engineer