Watson Curation Playback 0 - Draft v8 2

ECM Design Studio . confidential

Playback -1 v2

IBM Watson Curator May 14, 2015

2IBM Watson Curation : The Team (internal stakeholders)

the team

Product Management

Stephen Hussey -Program Director, Product Management

Jos Olminkhof -Product Manager, Product Management

Designers

Dana Morris -Program Director, ECM UX Design Tina Adams -UX Design Lead and Visual Designer Stacia Sambar -Information Designer

Product Development

Mike Marin -Distinguished Engineer, Case Manager Architect, US CA Oren Paikowsky -SW Development Manager, Israel Josemina Magdalen -ECM Content Discovery & Classification Product Architect, Master Inventor, Israel Manjula Hosurmath -Sr Engineering Manager, India Seema Meena -Developer, India Vishnu N KV -Developer, India Ravi Ray -Developer, India Sathis Marudanayagam -Tester, India

Sunil Satti -Developer, India

3IBM Watson Curation : Mission

Mission

Watson Curator enables Customers to deliver relevant, qualified and governed information collections for Watson

4

Technical Foundation

• New Services and architecture changes

•Auto Classification – Content Classification

•Document transformation– Datacap

•Deployment acceleration – Docker

•Version to version update/migration

• Standards and certifications • Globalization • FISMA / ISO (27000/27001) • Accessibility • SaaS operating support for Federal • Regulated data • FIPS 140-2 • IPV6

Federal Compliance factors Status and mitigation plan for any factor not compliant

Will the offering claim FISMA compliance? Will target to be FISMA ready by 06/30/2015

Is the offering accessible as per Section 508 of the Americans with Disabilities Act (ADA)?

Will target to be 508 compliant by 06/30/2015

Does the offering support compliance to FIPS 140-2 (Cryptographic Components)? Will target to be FIPS 140-2 compliant by 06/30/2015

Does the offering support IPV6? Will target to be IPV6 compliant by 06/30/2015

The request was reviewed and approved by Steve Mills on 11/11/2014Date deviation approval was granted by Steve Mills – 11/11/2014

Watson Curator

Curator Collection

Curator Collection

Curator Collection

Curator Collection

Curator Collection

Curator Collection

Curator Collection

Curator Collection

Curator Collection

WatsonEngagementAdvisor (WEA)

UnstructuredData

Curator Mission5

Team Lead

Watson Deployment Lead, provides

information criteria

Content Curator

Manages information collections and assigns

work items

Data Expert

Locates unstructured content

Domain Experts

Classify, review and approve information

Curator roles

Curator (and Domain Expert)7

Domain Expert (part time)8

Compelling reasons for a new Watson Curator releaseClaim Who says that? Priority?

1 Source documents are not easily consumed by Watson Engagement Advisor. They are often in the wrong format and require formatting

‘Project Jewel’ and other customers

high

2 The current product does not comply with federal industry standards that are considered important for this type of solution

IBM mandates high

3 The curation tasks are as not self-explanatory and intuitive Customers / Prospects high

4 Collaboration is crucial for efficiency and accuracy, but the user interface does not provide enough capabilities to collaborate

‘Project Jewel’ high

5 Source documents are not easily consumed by Watson Engagement Advisor. The text format requires editing, e.g. most are in declarative format, they all need to be in conversational format.

‘Project Jewel’ high

6 Executing curation tasks requires too much effort and is time consuming Customers / Prospects high

10

Year 2015

Hills

3 Katherine a Curator can:

easily locate, transform and analyze data of any kind into useable information

in a single collection


optimize curation tasks and activities to help the system improve automatically

resulting in a reduction of effort


recruit a team, set goals, assign engaging and intuitive curation tasks and activities in a way that suits the way they work

and become more efficient over time

who

what

WOW

Product visual framework will ingest IBM ‘Galaxy’ which is currently under visual refresh by Q3/4 2015.

What we will show around foundational look & feel is an approximation of a snapshot in time from Galaxy and will

evolve as Galaxy solidifies their UI.

We will present: As-Is

(current product UI)To-Be

(general idea of the UI as Galaxy develops & Dev resources allow)Future

(robust UI as Dev resources allow)

12

Hill 1Katherine a Curator can: optimize curation tasks and activities to help the system improve automatically resulting in a reduction of effort.

13Katherine a Curator can optimize curation tasks and activities to help the system improve automatically resulting in a reduction of effort

As-isEntirely manual processing, routing and classification effort as part of Curator and SME tasks

The need Reduce the human effort to consistently curate documents over the life of the collection

1

14

the scenario:Katherine, a Curator, is tasked with creating a new collection called Cancer;

• to be populated with a minimum of 6000 documents• 7 classification categories containing min 2000 documents• covering different area's of expertise• with reasonable accuracy level settings for auto classification

As a result of enabling auto classification, the SME, over time, will see a significant reduction in the number of documents they will need to classify, and Katherine will have some detailed metrics with which to base her human resource and document decisions upon.

Let’s look at Katherine’s experience first!

1

1As-Is: Curator My Collection view

Katherine logs in and selects the Add CollectionButton.

The look and feel will change when Galaxy develops their new UI which we will adopt. New screens are showing the Galaxy direction.

1To-Be: Curator My Collection view

17

1FUTURE VIEW: Curator My Collection view

Cancer

2,487

2

597Collected

675Working

2,487Curated

870Excluded

completenessbased on collection

details

alerts

collectionteam(Curator, SMEs, Project Lead, Data Expert)

analytics

18

1As-Is: Curator: Add New Collection View (no auto classification in current product)

19To-Be: Add Collection Screen SINGLE VIEW

IBM Watson CuratorKatherine Curator

Collections Work Add Collection

Collection Details

* Collection name:

Collection ID:

Expiration date:

Attachments:

Description:

Scope:

Purpose:

No file selectedBrowse File 1.pdf File 2.pdf

Clear

Cancer

C-10010234

This collection contains all types of human cancer.

Scope text here

This collection is to be used by Clinicians and Researchers in

Cancer diagnosis and treatment.

11 / 24 / 16 12:00 pm EMT

Expert Groups

Name Description

Stanford Medical

UCLA Medical

Stanford cancer specialists

UCLA cancer specialists

Topics

Classifications

Define classification types that will assist users in properly classifying documents in the collection.For each classification type, you can also add one or more classification values.

Categories Values DescAutoclassification Confidence levelsDesc

Enter new classification tvalue here

Drugs

Radiation

Chemotherapy

Eastern Medicine

Treatment

Cancer Type

Symptoms

Diagnosis

OFFON

OFFON

OFFON

OFFON

Min Docs/category: 200067%



Enter new category here

1

Katherine enters in all of the pertinent collection data.

She enables auto classification for all categories with the exception of Symptoms, at the requester’s guidelines.

Katherine adds values to each Category

1To-Be: Add Collection Screen ALTERNATE TABBED VIEW 1



ClassificationCollection Details Topics Expert GroupsCollection Details

* Collection name:

Collection ID:

Expiration date:

Attachments:

Description:

Scope:

Purpose:

No file selectedBrowse

File 1.pdf

File 2.pdf

Clear

Cancer

C-10010234

This collection contains all types of human cancer.

Scope text here

This collection is to be used by Clinicians and Researchers in

Cancer diagnosis and treatment.

11 / 24 / 16 12:00 pm EMT

ON

Minimum Included:

Collection Frequency:

Activity Reports:6,000 Daily AM

Daily AM




ClassificationCollection Details Topics Expert GroupsClassifications

Define classification types that will assist users in properly classifying documents in the collection.For each classification type, you can also add one or more classification values.

Categories Values DescAutoclassification Confidence levelsDesc

Enter new classification tvalue here

Drugs

Radiation

Chemotherapy

Eastern Medicine

Treatment

Cancer Type

Symptoms

Diagnosis

OFFON

OFFON

OFFON

OFFON




Enter new category here




ClassificationCollection Details Topics Expert GroupsExpert Groups

Name Description

UCLA Medical UCLA cancer specialists

Stanford Medical Stanford cancer specialists

Jana Jenkins

Bob Piercy

Beth Owens

408-555-1212

408-333-1212

408-444-1212

Cancer Diagnostics Specialist

Head of Stanford Cancer Department

Stanford Cancer Resident Physician

[email protected]

[email protected]

[email protected]

Enter new group member here

Find me experts

Send an invitation

23

Let’s see how autoclassification metrics can help Katherine make fast and easy decisions as she does

her daily job.

Value auditingWorkload rebalancing

1

24

1To-Be: Value Auditing

Katherine can see:• all the categories

at a glance• previous

accuracy level adjustments

• system suggestions to merge categories

• underperforming values

• when values have reached the collection minimum

• when values are under the minimum (grey)

• make inline auto classification adjustments without having to leave the screen

• see popup of detailed descriptions of each category and value


Collections Work Value Auditing

Cancer

Bone

Colon

Esophageal

Gall Bladder

Ovarian

Liver

Skin

Cancer Type (7)

Treatment

Symptoms

Diagnosis

Drugs

Radiation

Chemotherapy

Easterm Medicine

2310

2004

100

2032

OFFON Min Docs/category: 200090%


OFFON


2025

1500

270

328

2570

140

370

25

1FUTURE VIEW: Value Auditing alternate deep dive

IBM Watson CuratorKatherineCurator

Collections Work

Treatment | Drugs

Treatment | Radiation

Treatment | Chemotherapy

Treatment | Eastern Medicine

Treatment | Surgery

Treatment | Alternative

February March April May

Today

1overlappingvalues pairs

1accuracy leveladjustments

20sources

Projected Value goal 5/17/15 2,000 documents/value Included

2000Min docs/value

67%

projected min docs date based onprevious ingestion metrics

accuracy level

Cancer Value auditing projection

1underperforming

values

26

1FUTURE VIEW: Value Auditing alternate deep dive

IBM Watson CuratorKatherineCurator

Collections Work

Treatment | Drugs

Treatment | Radiation

Treatment | Chemotherapy

Treatment | Eastern Medicine

Treatment | Surgery

Treatment | Alternative

February March April May

Today

1overlappingvalues pairs

1accuracy leveladjustments

20sources

Projected Value goal 5/17/15 2,000 documents/value Included

2000Min docs/value

67%

projected min docs date based onprevious ingestion metrics

accuracy level

Cancer Value auditing projection

ChemotherapyActions

DrugsMerge with:

Adjust: Minimum documents/value: 200

CancelSave

1underperforming

values

27

1To-Be: Workload viewing/rebalancing

Katherine can now immediately see potential problem areas at a glance.• overdue• unassigned• assigned

She is able to quickly reassign tasks to different groups or specific SMEs.

She can quickly assign tasks or ingestions to groups or specific SMEs.

IBM Watson CuratorJanet BauerCurator

15

My Collections

597Collected

675Working

2,487Curated

870Excluded

Domain Expert would not see Collected click this and

go to details

Curation task 1/22/15





300 documents

100 documents

20 documents

500 documents

50 documents

UCLA Medical

UCLA Medical

UCLA Medical

UCLA Medical

Overdue (970) UCLA Medical

3/23/15

3/23/15

3/23/15

3/23/15

1:00 pm GMT

9:00 am GMT

10:00 pm GMT

4:00 pm GMT

Unassigned (525) UCLA Medical

Assigned (473) UCLA Medical

Ingestion 3/12/15

Ingestion 3/17/15

Ingestion 4/1/15

Ingestion 4/3/15

ingestion 4/4/15

200 documents

100 documents

50 documents

75 documents

100 documents

By status

By taskBy groupBy personBy date dueBy status

Cancer

Stanford MedicalDue: Assign to:5 / 24 / 15 12:00 pm EMT

Stanford MedicalDue: Assign to:5 / 20 / 15 2:00 pm EMT

Group/personDue: Assign to:5 / 24 / 15 12:00 pm EMT




28

Now let’s look at the SMEs experience!

1

29

1FUTURE VIEW: SME Dashboard view

30

1FUTURE VIEW: SME Dashboard view

1FUTURE VIEW: SME My Work idea

This could be a possible view into My Work as an SME.

This idea also shows collaboration:• query

colleague for a review

• send a note to team member

• redirect a document to a team member

• make personal notes on a document

IBM Watson CuratorJanet BauerSME

My Work

15

Cancer

Tumors

Viral Disseases

Motaba

0

5,000

4,207

Liver Disseases

4,207

2,487

Recent Collections 597Collected

675Working

2,487Curated

870Excluded

Bone_Metastasis.pdf32 pages

17 pages

48 pages

11 pages

Added by DSanderson on 1/14/2015 at 3:02 pmSource: IBM Stored IQModified by DSanderson on 1/15/2015 at 11:57 am

Today at 1:39: to @Susan Longhi: @Stephen Hussey, I need your opinion on this document.

12/29/14 at 2:54: from @Susan Longhi: I took a look at it and think we may need to call in another Domain Expert due to the nature of this document. I have seen...

Today at 1:42: need to follow up on this source, alerted Stephen & Susam.

Yesterday at 3:37: unsure of this source

Cancer_treatment_study.pdf


Melanoma_study.pdf

Added by DSanderson on 1/14/2015 at 3:02 pmSource: IBM Stored IQ

Source: IBM Stored IQ

Modified by DSanderson on 1/15/2015 at 11:57 am

Lymphoma_study.pdf

Added by DSanderson on 1/14/2015 at 3:02 pm


12/29/14 at 2:54: @Curator: This file is corrupted, so I think we nee to transform again.

12/29/14 at 2:54: #reassign@Seema Meena: I need you to classify this document based on your expertise

V

by urgency

by priorityby urgencyby activityby name

my notifications my notes





Domain Expert would

not see Collected

Could also have comments and notes on a TASK level too!!!!

Search

click this and go to details

1To-Be: SME Classification Screen (autoclassification has just been enabled, no suggestions yet)

When the system first starts to learn, it cannot make any suggestions for classification to the SME yet.

IBM Watson CuratorJanet BauerDomain Expert

My Work | Cancer | Document_1.html

15

Cancer_1002.pdf: 67% confidence

Classification Summary Description Statistics Collection Goals

Search in document

Include Exclude

2

Previous | Next

View similar documents in the collection

Search

Submit new

Bone

Ovarian

Colon

Espphageal

Gall Bladder

Liver

Skin

Search

Biopsy

Blood Test

X-Ray

CAT Scan

PET scan

Spinal Tap

Ultrasound

Search

Submit new

Submit new Submit new

Drugs

Radiation

Surgery

Eastern medicine

Search

Submit new

Exhaustion

Immune disorders

Infection

Mobility

Pain

Rash

Weakness

Search

Stage 0

Stage 1

Stage 2

Stage 3

Stage 4

End-Stage

Prognosis Diagnosis

SymptomsSymptomsTreatment Type Cancer Type

Autoclassification just turned on-> No suggestions yet

1To-Be: SME Classification Screen (autoclassification has just been observing over time)

As an SME, we now can see that Curator is making suggestions for us to look at and consider.



15


Classification Summary Description Statistics Collection GoalsInclude Exclude

2

Previous | Next


Search

Submit new

Bone (30%)

Ovarian (30%)

ColonEspphagealGall BladderLiverSkin

Search

Biopsy (55%)

Blood Test (46%)

X-Ray (22%)

CAT ScanPET scanSpinal TapUltrasound

Search

Submit new


Drugs (30%)

Radiation (28%)

Surgery (25%)

Eastern medicine

Search

Submit new

ExhaustionImmune disordersInfectionMobility PainRashWeakness

Search

Stage 0 (37%)

Stage 1 (35%)

Stage 2 (33%)

Stage 3 (33%)

Stage 4 (21%)

End-Stage

Prognosis (77%)Diagnosis (33%)

SymptomsSymptomsTreatment Type (72%)Cancer Type (40%)

Autoclassification is learning and making suggestions.

1To-Be: SME Classification Screen (autoclassification is running smoothly)

We now see that Curator has reached it’s stride and has provided us with confidence levels and solid category value choices, allowing us to be able to quickly make decisions and improve the accuracy of the classification.

In the future, could show ground truth WEA mapping overlay for docs that need to be re-curated.



15


Classification Summary Description Statistics Collection Goals

Search in document

Include Exclude

2

Previous | Next


Search

Submit new

Bone (55%)

Ovarian (30%)

ColonEspphagealGall BladderLiverSkin

Search

Biopsy (100%)

Blood Test (88%)

X-Ray (58%)

CAT ScanPET scanSpinal TapUltrasound

Search

Submit new


Drugs (100%)

Radiation (89%)

Surgery (44%)

Eastern medicine

Search

Submit new

ExhaustionImmune disordersInfectionMobility PainRashWeakness

Search

Stage 0 (97%)

Stage 1 (88%)

Stage 2 (70%)

Stage 3 (47%)

Stage 4 (32%)

End-Stage

Prognosis (77%)Diagnosis (33%)Diagno

SymptomsSymptomsTreatment Type (72%)Treatment Type (7Cancer Type (40%)Cancer Ty

Autoclassification has learded after a time and is making confident suggestions

1As-Is: Curator: collection properties dialog

36

Hill 2Katherine a Curator can: recruit a team, set goals, assign engaging and intuitive curation tasks and activities in a way that suits the way they work and become more efficient over time.

37Katherine a Curator can: recruit a team, set goals, assign engaging and intuitive curation tasks and activities in a way that suits the way they work and become more efficient over time

As-isCurator team engagement is manual and repetitive.SME context, inputs and work management is limited. The experience requires training

The need Optimize the Curator’s use of a team Present the SME task experience in an optimized layout for information and actions

332

38

Overview of the To Be process34

Curator

Select itemFrom

inbasketClassifyand Tag

Include/Exclude Repeat

SME

Administer classification categories as

needed

Tune classification to

reduce effort

Assign work to SME

As Katherine creates a new collection, she specifies a scope and purpose for the collection, helping the entire team to stay focused.

Create new collection

Initial config auto

classification

Curate for xxx days/

weeks

The SME collaborates with Katherine, and with other SMEs, reassigns the curation task, or sub-task(s) for individual documents in the task, and saves time by leveraging the expertise of other SME's.

With a few mouse clicks, Katherine then easily assigns a task to a SME requesting to review several documents to be curated.

2

39

the scenario:Katherine, a Curator, needs to assign and manage SME groups in her new collection called Cancer;

• to

As a result, SMEs….

Let’s look at Katherine’s experience first!

1

1To-Be: Curator Expert Groups assignment & management

Katherine creates expert groups and starts to assign members.

In the future, Curator could keep track of all the various SMEs and make suggestions based on specifics defined in Collection Details.

In the future, Katherine could track SME availability and send a notice right from this screen.



ClassificationCollection Details Topics Expert GroupsExpert Groups

Name Description

UCLA Medical UCLA cancer specialists

Stanford Medical Stanford cancer specialists

Jana Jenkins

Bob Piercy

Beth Owens

408-555-1212

408-333-1212

408-444-1212

Cancer Diagnostics Specialist

Head of Stanford Cancer Department

Stanford Cancer Resident Physician

[email protected]

[email protected]

[email protected]

Enter new group member here

Find me experts

Send an invitation

41

Curator can view Document analytics in a collection and adjust based on time, priority, and SME

2FUTURE VIEW: Curator Collection Analytics view

Katherine is able to assess workload and able to shift tasks between either people or Expert Groups.

42

2FUTURE VIEW: SME Work screen: Collaboration

IBM Watson CuratorJanet BauerSME

My Work

15

Cancer

Tumors

Viral Disseases

Motaba

0

5,000

4,207

Liver Disseases

4,207

2,487

Recent Collections 597Collected

675Working

2,487Curated

870Excluded

Bone_Metastasis.pdf32 pages

17 pages

48 pages

11 pages


Today at 1:39: to @Susan Longhi: @Stephen Hussey, I need your opinion on this document.

12/29/14 at 2:54: from @Susan Longhi: I took a look at it and think we may need to call in another Domain Expert due to the nature of this document. I have seen...

Today at 1:42: need to follow up on this source, alerted Stephen & Susam.

Yesterday at 3:37: unsure of this source

Cancer_treatment_study.pdf


Melanoma_study.pdf

Added by DSanderson on 1/14/2015 at 3:02 pmSource: IBM Stored IQ

Source: IBM Stored IQ


Lymphoma_study.pdf

Added by DSanderson on 1/14/2015 at 3:02 pm



12/29/14 at 2:54: #reassign@Seema Meena: I need you to classify this document based on your expertise

V

by urgency

by priorityby urgencyby activityby name






Domain Expert would

not see Collected

Could also have comments and notes on a TASK level too!!!!

Search

click this and go to details

SME can see message/collaboration threads about specific documents and perform actions such as reassign to a colleague in their collection team.

SME can also make personal notes on a specific document as well.

43

Hill 3Katherine a Curator can: easily locate, transform and analyze data of any kind into useable information in a single collection.

44Katherine a Curator can: easily locate, transform and analyze data of any kind into useable information in a single collection

As-is

Valuable information is inaccessible to Watson. E.g. PDF and Word files with multi column table based information are inaccessible without manual editing. Data with inconsistent layouts and formatting are extremely challenging to use.

The need

Auto transform information like PDF and Word into well structured HTML for WEA consumption

3

45

Overview of the To Be process

41

Curator

Select itemFrom

Inbasket

Classifyand Tag

Edit Document or

VersionInclude Repeat

SME

Curate for xxx days/weeks

TransformTo HTML

PDF

DOC

(XLS)

Compare wprevious version Transform text

for WEA consumption

Base formats supported:

Linear format / Normalized

Changes are highlighted

New document

New versionAlways / On demand

3Katherine a Curator can: easily locate, transform and analyze data of any kind into useable information in a single collection

46

Curator can view Document analytics in a collection and adjust based on time, priority, and SME

FUTURE VIEW: SME version compare

SME is able to easily see where WEA mapping exists and changes were made between versions.

IBM Watson CuratorJanet BauerAll rolest


15

77% confidence

can we do a bg diff and only show those areas that are different? those areas that are mapped in gt?

Supported:word, pdf, htm

Transforn:orig vs transformedability to approve transform,or rejectl

version 1.0 version 2.0

accepting replaces the curren v1.0with this version, v1.0 moves to Excluded

rejecting keeps v1.0 in the collection,v2.0 moves to Excluded

Visual Diff buttonEdit button

3

47

Project dates, milestones, and details…

48

Timeline

Q1 Q2 Q3 Q4Change Management

Q3 Q4

Data transformation and extraction

Auto classification

User efficiency

Q1 Q2

Milestones for V2

Hills Playback 8 April

Playback 0 1 May

Playback 1 27 May

Playback 2 24 June

Playback #...22 July V2 Availability

Offering Business

Plan5th May

Product ReadinessPMOM

49

~4 Weeks

User research and sponsor usersValidating assumptions

Org Type Contact Hill NDA Approached Confirmed Engaged

Geico Customer Jack Jones 1 and 2 N Y N N

IBM Consultant Sridhar Sudarsan 3 N/A N N N

IBM Sales Maig Worel 1,2,3 N/A Y Y N

50

Backup

51

Feedback

• Huzefa – The variable timeline

•Challenges with go to market activities. We will have to figure out the lead time required to trigger GTM and line it up with the last playback

• Jos – Product dependencies

• Introducing final delivery uncertainties across teams will have to be managed.

• Manjula – Migration

•Add version to version update/migration to the Foundation

52

To-be: Curator Create New Collection View: with scope and purpose 2

Documents

Watson Curation Playback 0 - Draft v8 2