1 Copyright © 2011, Oracle and/or its affiliates. All rights€¦ · Including Duplicates and Missing Information. Product ID Manufa-cturer Description Product Type Power Voltage

1 Copyright © 2011, Oracle and/or its affiliates. All rights

reserved.Confidential – Oracle Restricted



Universal Approach to Data Quality

with Oracle Enterprise Data Quality

Martin Boyd, Senior Director, Enterprise Data Quality

Mala Narasimharajan, Senior Product Marketing Manager Session ID: 19400


reserved.

The following is intended to outline our general product

direction. It is intended for information purposes only, and may

not be incorporated into any contract. It is not a commitment to

deliver any material, code, or functionality, and should not be

relied upon in making purchasing decisions.

The development, release, and timing of any features or

functionality described for Oracle‟s products remains at the sole

discretion of Oracle.


reserved.

Agenda

• Why Bother With Data Quality?

• Data Quality Domains – Does One Size Fit All?

• “Fit for Purpose” – Measuring & Assuring High DQ

• Enterprise Data Quality – Product Overview

• Data Quality Application: Watchlist Screening


reserved.

© 2009 Oracle Corporation© 2009 Oracle Corporation

Your Data is Growing

It is estimated that 4 exabytes (4.0X1018)

of unique information will be generated this year.

That is more than the previous 5000 years.


reserved.

Companies

Your Data is Changing

• 240 businesses will change addresses

• 150 business telephone numbers will change or be disconnected

• 112 directorship (CEO, CFO, etc.) changes will occur

• 20 corporations will fail

• 12 new businesses will open their doors

• 4 companies will change their name

Source: D&B, US Census Bureau, US Department of Health and Human Services, Administrative Office of the US Courts,

Bureau of Labor Statistics, Gartner, A.T Kearney, GMA Invoice Accuracy Study

• 5,769 individuals in the US will change jobs

• 2,748 individuals will change address

• 515 individuals will get married

• 263 individuals will get divorced

• 186 individuals will declare a personal bankruptcy

Individuals

Master data changes at rate of 2% per month

Products

• On average 20% duplicates in product data

• 90% product introductions fail

• Retailers lost 40 billion or 3.5% of total sales lost each year due to item info inefficiencies

• 60% error rate for all invoices generated

• Global Data Sync will realize 30% lower IT costs

In one hour… In one hour… In one year…

Compounded, 2% monthly change is 27% per year, 61% in two years, 104% in three years!!!


reserved.

Your Data Contains Errors and Inconsistencies

Variation or Error

ExampleVariation or

ErrorExample

Sequence errors • Mark Douglas or Douglas MarkTranscription

mistakes• Hannah, Hamah

Involuntary corrections

• Browne – BrownMissing or extra

tokens• George W Smith, George Smith, Smith

Concatenated names

• Mary Anne, MaryanneForeign sourced

data

• Khader AL Ghamdi, Khadir A.

AlGamdey

Nicknames and aliases

• Chris – Christine, Christopher, TinaUnpredictable

use of initials• John Alan Smith, J A Smith

Noise• Full stops, dashes, slashes, titles,

apostrophesTransposed

characters• Johnson, Jhonson

Abbreviations• Wlm/William, Mfg/Manufacturing Localization • Stanislav Milosovich – Stan Milo

Truncations • Credit Suisse First Bost Inaccurate dates• 12/10/1915, 21/10/1951, 10121951,

00001951

Prefix/suffix errors

• MacDonald/McDonald/DonaldTransliteration

differences• Gang, Kang, Kwang

Spelling & typing

errors• P0rter, Beht Phonetic errors • Graeme – Graham


reserved.

With More Variations Than You Can Imagine

10hp motor 115V Yoke mount

mtr, ac(115) 10 horsepower 115volts

MOT-10,115V, 48YZ,YOKE

This 10hp yoke mounted motor is rated for

115V with a 5 year warranty

10 Caballos, Motor, 115 Voltios

TEAO HP = 10.0 1725RPM 115V 48YZ YOKE MTR

Motor, TEAO, 1725 RPM, 48YZ, 15 Voltios,

Montaje de Yugo, hp = 10

Item Motor

Classification 26101600

Power 10 horsepower

Voltage 115

Mounting Yoke


reserved.

Including Duplicates and Missing Information

Product IDManufa-

cturerDescription Product Type Power Voltage Mount

ABC123 AA Inc. 10hp motor 115V Yoke mount Motor, AC/DC 10hp 115V Yoke

abc-123 A.A. mtr, ac(115) 10 horsepower 115volts AC/DC Motor 10 115 AC.

ABC/123/Q AA/Craft 10 Caballos, Motor, 115 Voltios Mot-AC 10 H-pow 115

QA-ST5 Craft TEAO HP = 10.0 1725RPM 115V 48YZ YOKE MTR 26101604

Z99 Z99 MOT-10,115V,48YZ,YOKE Z99

Motor powered pulley, 3/4" Motor

Inconsistent names

(representing same business)

(Often) Rich information, but

mostly non-standard

Attributes non-standard,

missing or invalid

Mis-classified item

(not a motor)

Companies struggle with the basics of Product Data

• 80% companies are not confident in the quality of their product data

• 73% find it “difficult” or “impractical” to standardize product data „PIM Business & Technology

Trends - Survey‟, Sept 2007

Inconsistent formats (extra

characters often added)

Inconsistent classifications &

misclassifications

Widespread

duplication

(often hard

to spot)


reserved.

Business Impact of Data Quality

With Bad Data With Good Data

• Reduced ROI

• Increased project risk, time and cost

• Expensive downstream consequences

– wrong shipment, wrong invoices,

incorrect parts…

• Increased ROI on existing systems

• Increased agility

• Increased efficiency

• Increased customer satisfaction

• Increased scalability

“Only 30% of BI/DW

implementations fully succeed.

The top two reasons for failure?

Budget constraints and data

quality.”

“Data integration and data quality are

fundamental prerequisites for the

successful implementation of enterprise

applications, such as CRM, SCM, and

ERP.” ”

“#1 reason CRM projects fail:

Data Quality”


reserved.

Data Quality Delivers Value Across All Industries• Single view of high quality customer data drives accurate customer insight and

improved marketing effectiveness

• Supports compliance and reporting KYC requirements

• Single view of citizen for better internal information sharing, service delivery, licensing, provision of child care, and fraud detection

• Reduce costs through system rationalisation

Financial Services

• Harmonizes customer data from multiple channels to improve sales and marketing effectiveness

• Enhance online product search for ECommerceRetail

• Improves customer insight for revenue optimization and targeted customer retention

• Effective compliance and risk mitigation for next generation servicesTelco

Energy & Utilities• Expands understanding of network assets and customer delivery points

• Improves management of regulatory compliance and reporting requirementsUtilities

• Delivers a comprehensive view of patient for care and billing

• Manages patient, epidemiology, diagnosis and treatment data quality across systems and organizations

Healthcare

Government


reserved.

Agenda







reserved.

Many Domains, Many DQ Problems

Party

Data

Product

Data

All

Data

Customer

Employee

Citizen

Patient

Operational

Supplier

Locations

Financial

Sales

Forecast

Data Warehouse

AssetsItems for

Sale

MRO SKUs

ItemProcurement Items

Core Requirements:

• Name and address

verification, standardization,

match

• Multiple languages & locales

• Locale-specific processes

Core Requirements:

• Multi-category architecture

• Semantic recognition and

learning

• Attribute extraction and

standardization

• Classification

• Translation

Core Requirements:

• Easy to configure based on data

• Transparent/tunable rules

• Share and reuse custom processes

• Immediate feedback based on data


reserved.

Customer Data Quality vs. Product Data QualityDifferent problems, different approaches

Party Data Product (part, SKU, item, asset etc.)

Over 30,000 different product categoriesVariations by

country/locale

Require:

• Pre-built processes

• Pattern-based matching

Name & Address Data

• Relatively fixed syntax

• Mis-spellings and name equivalents

• Verify against postal files

Require:

• Semantic understanding

• Ability to learn new domains quickly

Product Data

• No fixed syntax – few standards

• Infinite variability – format, content, syntax

• Different rules for each product category


reserved.

Agenda







reserved.

• How do you know?

• What is the business impact?

• What should you do about it?

Data Quality – Is Your Data “Fit for Purpose”?


reserved.

Health Check – Is Your Data “Fit for Purpose”?

• Understand current data „fitness for purpose‟

• Estimate DQ impacts & ROI

• Identify critical issues & quick wins

Understand

Improve

Protect

GovernYour

Data

Your Experts

Current

issues,

gaps,

errors

Business &

data

standards


reserved.

Improve Data, Improve App Performance

• Improve ROI and performance of existing applications

• Engage users and executives

• Bring data to a known, baseline quality – ready to roll-

out new applications and initiatives

Understand

Improve

Protect

Govern

Metrics,

KPIs

Fit for

purpose

data

Parse/

extract

Stand-

ardize

Match/

merge

Verify

Enrich

‘Gold’

data

Apply data standards


reserved.

„DQ Firewall‟ – Continuous Protection for Information Assets

• Continuous, consistent enforcement of standards

• High quality data drives ROI

• No more DQ projects!

Understand

Improve

Protect

Govern

Hub

Apply data standards/validate

External

sources/

feeds

Data Integration/ETLNon-DQ/MDM-

aware Apps

DQ/MDM-

aware AppsWeb

service

call


reserved.

DQ Governance – Continuous Process Improvement

• Monitor ongoing effectiveness

• Track and resolve issues

• Improve overall effectiveness

Understand

Improve

Protect

Govern

DQ/MDM-

aware Apps

Target

system DQ

metrics

‘Gold’

data

Apply data standards

Source

system DQ

metrics

DQ

process

metrics


reserved.

Agenda







reserved.

Introducing Enterprise Data Quality

DQ-Based Solutions

Domain Knowledge

Business Solutions• Customer-delivered

• Partner-delivered

• Oracle-deliveredApplication Connectors

Data Quality Platform

• Complete range of DQ capabilities

• Best-of-breed capabilities for party and product

data

• Easy to use, intuitive

• Open, tunable, flexible

Pre-Built Solutions

• Any scope – components to end-to-end solutions

• Any pre-built/reusable item– Processes, methods

– Knowledge, reference data

– Application integration

Enterprise Data Quality

Dashboards

Party Data

Extensions

Match/Merge

Governance

Product Data

Extensions

Standardization

Profile and Audit


reserved.

Introducing Enterprise Data Quality

DQ-Based Solutions

Domain Knowledge




Benefits

• High quality data boosts ROI of any application

or technology („bad‟ data limits ROI)

• Complete solution with best of breed

capabilities for both Party and Product entities

• Essential component in any Data Integration or

Master Data Management (MDM) strategy

• Avoid doing business with sanctioned or risky

individuals and entities


Dashboards

Party Data

Extensions

Match/Merge

Governance

Product Data

Extensions

Standardization

Profile and Audit


reserved.

Enterprise Data Quality – for Party Data

DQ-Based Solutions

Domain Knowledge




Party Data Capabilities

– Pre-built processors – to handle most

common name & address problems

• Address parsing & standardization

• Address verification & geocoding*

• Match/merge – individuals, households, businesses

• Country, language & locale-specific functions

– International Reference Data – for global

deployment

– Integration with Oracle Customer Hub

(UCM) – for rapid time to value

* Available soon


Dashboards

Party Data

Extensions

Match/Merge

Governance

Product Data

Extensions

Standardization

Profile and Audit


reserved.

Enterprise Data Quality – for Product Data

Product Data Capabilities

– Semantic recognition – to deal with extreme

variability of product information

– Category-specific architecture – to deal with

large number of different product categories

– Pre-built knowledge – for common product

data attributes

– Integration with Oracle Product Hub – for

rapid time to value

DQ-Based Solutions

Domain Knowledge





Dashboards

Party Data

Extensions

Match/Merge

Governance

Product Data

Extensions

Standardization

Profile and Audit


reserved.

DQ Spans MDM and Data Integration

Custom

Applications

Watchlist

Screening

Oracle

Applications

3rd Party

Applications

Oracle

BI/EPM

Content ManagementServices

Business IntelligenceServices

CollaborationServicesData Services

Oracle

MDMOracleData

Integration

Data Federation Replication

Customer Hub Product Hub

Information Management

OLTPSystem

Data Warehouse/Data Mart

OLAP Cube

Supplier Hub Financial Hub

Storage

Transformation SynchronizationETL/E-LT

Site Hub

Web and Event Services, SOA

Transaction ProcessingServices

Enterprise Data QualityProfiling Standardization Match/Merge


reserved.

Su

pp

lie

r

Sit

e

Fin

an

cia

l

Enterprise-Grade MDM from OraclePurpose-Built MDM Applications Approach

CRM

Other Sources

ERPC

usto

me

rCRM

EPM/BI

Legacy…

Pro

du

ct


reserved.

EDQ With Oracle Data Integrator: Use CasesSources

Target(s)

E.g. Data

Warehouse

such as Exadata

Oracle Data

Integrator

One-off ProfilingUnderstand data to build

ODI transformation

and mapping processes.

Automated ProcessesDe-duplication, complex

transformation and parsing

called during ODI data flow.

Measure Ongoing DQAssess quality of data

in target system. How well

is ETL working?

Enterprise Data

Quality


reserved.

EDQ and ODI: Complimentary Features

Oracle Data Integrator Enterprise Data QualityExtracts, Transforms, Loads Extracts, Transforms, Loads

'Majors' on extraction and loading Basic extraction and loading capabilities, but merely a means to an end

May also be used to perform ‘simple’ transformations:• Concatenate • Simple parsing • Remove invalid values

Strong matching, transformation and parsing capabilities:• Parse & standardize address fields• Parse & standardize product description• Extract & standardize key attributes• Locale-specific rules• Categrory-specific rules etc.

Use strong profiling capabilities to design ODI processes.

E t L e T l


reserved.

Main EDQ Console, Focused on the User

ProjectBrowser

Main Canvas

Results Browser

Tool Palette


reserved.

Immediate drill-down to examine real data

Field-level completeness, uniqueness, validity etc.


reserved.


Drill-down to see actual data values and determine

required rules, standards etc.




Drill-down to see actual data values and determine

required rules, standards etc.


reserved.

Build-out Full DQ Process

Process can be „simple‟ or „complex‟

Process thumbnail assists in navigation


reserved.

Key Feature: Pre-built processors

• Comprehensive DQ Functionality with a Single User Interface and Repository


reserved.

Key Feature: Language, Region and Country-Specific Capabilities

• Country and region-specific processors

• Fully unicode compliant


reserved.

Transformation – Data Improvement & Cleansing

Use profiling results to create your own data

improvement rules

Use provided processors for common tasks

such as address standardization

• Fully configurable data transformation rules

• Operates in both Batch and Real-Time

• Full control over data updates

• Original data always preserved (and all steps in between)

• Source data may either be staged and processed or „streamed‟ through the process


reserved.

„Cleansing‟ for Party Data

39

• Parse, standardize, transform

• Split names and name elements

• Identify individuals and businesses

• Derive additional attributes

Name: Dr Ellen Van Der Heijde

Title: Dr

First: Ellen

Last: Van Der Heijde

Gender: Female

Name: Jalila Abdul-Alim (Do Not Call)

First: Jalila

Last: Abdul-Alim

Gender: Female

Note: Do Not Call

Title: Mr

First: R

Middle: J

Last: MacDonald

Gender: Male

Name: Mr RJ & Mrs FB MacDonald

Title: Mrs

First: F

Middle: B

Last: MacDonald

Gender: Female

Title: Ms

First: April

Last: James

Gender: Female

Name: Ms April James DBA AJ Designs

Company: AJ Designs

Not just names but any data such as addresses, dates

& phone numbers etc.


reserved.

„Cleansing‟ for Product Data

Category

Piping & plumbing

Fasteners

Headgear

Capacitors

Wine

Extracted Attributes

Diameter = 4 inch

Length = 4 inch

Quantity = 4

Capacitance = 4ESU = 4.45 picofarad

Volume = 750ml; varietal = Merlot

Standardized Description

End cap, 4 inch

Hex cap screw, 4 inch

Baseball cap, 4 per box

Capacitor, 4.45pf (4 ESU)

Wine: Merlot, 750ml, screw cap

Missing information

Material; Thread

Thread; Diameter; Material

Color

Manufacturer; Mount type; Operating characteristics

Input Description

4” end cap

4 in cap screw

4 in box b-ball cap

4 ESU cap

750ml Merlot, screw cap

1. Semantic recognition uses whole context to determine meaning and category

2. Information extraction uses category-specific semantic models to identify and extract target information

3. Category-specific semantic models flag missing information for potential remediation

4. Data is converted, transformed and re-assembled according to requirements of target system

Semantic Rec.


reserved.

Matching – Duplicate Identification and Prevention• Designed for business users

• Flexible matching engine for any data with many comparison algorithms

• Provided template match processors for individual, entity and address matching

• Easy reuse of configured match processors

• Fully configurable outputs (Links, Groups, Master and Slaves, Best Record)

• Operates in both Batch and Real-Time

• See Match Essentials deck for more information on Matching

Pre-built rules can be

switched on and off and/or

customized


reserved.

Matching – Party Match Scenario

42

Title: Mr

First: Robert

Last: Fulmar

Gender: Male

DoB: 12/05/1978

Phone: 555-120-1329

Address:

9405 Main St

Fairfax

Virginia

22030

First: Bob

Last: Fulmar

Gender: Male

Email: [email protected] Title: Dr

First: R

Last: Fulmer

DoB: 01/01/1978

Email: [email protected]

Address:

9407 Main Street

Fairfax

VA

22031-4001

Title: Dr

First: Robert

Last: Fulmar

Gender: Male

DoB: 12/05/1978

Email: [email protected]

Phone: 555-120-1329

Address: 9407 Main St, Fairfax, VA

22031-4001

Match & Merge data from

disparate sources

Create the „best‟ record

Match Individuals

Match Households

Match Businesses


reserved.

Matching – Product Match Scenario

All Items

Needles

Biopsy Needles

Manufacturer MEDICAL DEVICE

TECHNOLOGIES, INC.

UOM BX

Item NEEDLE

Needle use BIOPSY

Needle type BONE MARROW

Diameter 11GA

Needle length 4IN

Point type

Needle style

Handle type

Reusability DISPOSABLE

MEDICAL DEVICE

TECHNOLOGIES, INC.

MEDICAL DEVICE

TECHNOLOGIES, INC.

MEDICAL DEVICE

TECHNOLOGIES, INC.

BX BX BX

NEEDLE NEEDLE NEEDLE

BIOPSY BIOPSY BIOPSY

BONE MARROW BONE MARROW BONE MARROW

11GA 11GA 11GA

4IN 4IN 4IN

DOUBLE DIAMOND DOUBLE DIAMOND

JAMSHIDI JAMSHIDI HARVEST

ERGONOMIC TWIST-LOCK ERGONOMIC TWIST-LOCK

DISPOSABLE

85% 70% 70%

Match #1 Match #2 Match #3

MEDICAL DEVICE

TECHNOLOGIES, INC.

MEDICAL DEVICE

TECHNOLOGIES, INC.

MEDICAL DEVICE

TECHNOLOGIES, INC.

BX BX BX

NEEDLE BIOPSY BONE

MARROW JAMSHIDI 11GA

4INL DISPOSABLE

NEEDLE BIOPSY BONE

MARROW JAMSHIDI

DOUBLE DIAMOND TIP

11GA 4INL ERGONOMIC

TWIST-LOCK HANDLE

NEEDLE BIOPSY BONE

MARROW HARVEST

DOUBLE DIAMOND TIP

11GA 4INL ERGONOMIC

TWIST-LOCK HANDLE

Manufacturer MD TECH

UOM BOX

Description NDL BONE MARROW

11G X 4IN THRWAWY

Potential PIM category matches

Required for

match

Raw

data

from

PIM

Optional -

weighted

Match score

Domain definitionsRecognition, validation,

vocabulary, relationships

etc.

Item from Hospital


reserved.

Dashboards – Snapshot & trend analysis

44

Validate data against business rules -

publish results to data quality dashboard


reserved.

Enterprise Data Quality Differentiators

Integrated DQSolution

Domain-specific capabilities

Ease of use

• Integration of all core DQ capabilities – profiling, cleansing,

classification, repurposing, matching & reporting

• Engineered for business users

• Easy to configure and integrate „DQ Services‟

• Integrated team collaboration and management

• Specialized capabilities for major domains – Party and Product

• Pre-built processes and knowledge

• Use „the right tools for the job‟ depending on the task

• Extensible for any domain, task, process

• Agnostic of data-domain, vertical market & application• Brings DQ out of the back-office• Users can monitor what matters to them

–Personalised dashboard

–Select content and define order

• Business context to gain understanding and consensus

45


reserved.

Agenda







reserved.

Oracle Watchlist Screening


MatchingData

Preparation

Case

Management


Dashboards

Party Data

Extensions

Match/Merge

Governance

Product Data

Extensions

Standardization

Profile and Audit

Broad leverage of EDQ platform* enhances Watchlist

Screening capabilities

Source data preparation

• Profile source systems

• Parse, standardize, transform data prior to matching

• Enrich and remediate as required

Matching

• Highly efficient, scalable match engine

• 500+ pre-built rules can be switched on and off

• Any custom rules can be built as required

• Fully internationalized data handling with – Name recognition

– Name and nickname standardization

– Cross-script transliteration, transcription for matching

Case Management

• Work queue management

• Alerts, reporting, audit trail


reserved.

Watchlist Screening• What it is:

– A business process for identifying a potential match between a customer record and anyone (customer, supplier, business partner, employee) that may appear a „watch list‟ (public or private)

• Why it matters:

– International and regional regulations

• Anti-Money Laundering (AML)

• Counter Terrorist Financing (CTF) legislation

– Determine the potential for exposure to money laundering, bribery & corruption and/or terrorist financing

• Pressure for a „next generation‟ solution:

– Improved results

• Better matching

• Reduced false positives

• Reduced operating costs

• Better scalability

48

“Know Your Customer” (KYC) and Enhanced Due Diligence (EDD) obligations

• Bank Secrecy Act 1970

• Foreign Corrupt Practices Act 1977

• Money Laundering Control Act of 1986

• Terrorism Act 2000

• Financial Services & Markets Act 2000

• USA Patriot Act 2001

• Proceeds of Crime Act 2002

• EU 3rd Money Laundering Directive 2007

• Money Laundering Regulations 2007

• UK Bribery Act 2010

• ...


reserved.

What Happens if it Goes Wrong?

• Substantial fines and

penalties against• Companies

• Individuals within companies

• Lloyds $350m

• CSFB $536m

• Aon £5.25m

• ANZ $7m

• ABN $500m

• UBS $100m

• Wachovia etc…

• Brand damage and loss of

reputation


reserved.

Office of Foreign Assets Control (OFAC) List

Approx 300,000

names just on

US OFAC list...


reserved.

Watchlist Screening Process

Customer lists• Multiple systems• Inconsistent information

& standards

Watch lists• Public lists (HMT, OFAC,

EU, UN, etc.)• Commercial Lists• Private lists

Understand

Structure

Standardize

Enrich

Translation/ transliteration

Aliases/name equivalents

Match & risk scoring

500+ match rules

Prepare & Optimize Match & Score Case Management

Prioritized alerts

Customizable workflow

Reporting & metrics

Audit trail


reserved.


Minimize Risk Exposure

Most FlexibleSolution

Lowest Total Cost of Ownership

• Low false positives due to underlying DQ technology

• Configurable risk & match scoring

• Designed for business user

• Complete audit trail of review process

• Optimizes all data for screening

• Integrated case management & reporting tools

• Agnostic to data format, language and structure

• 500+ match rules with option to customize

• Avoids unnecessary or repetitive investigations

• Screens all risk profiles

• Pre-built workflows enables rapid integration and deployment

• Low maintenance overheads


reserved.

Watchlist Screening Customer Proof Points

FirstRand*2

Barclays*3

Leading Investment Company

• 1m customer records screened

• False-positive rate = 5% of industry average*

• 10m customer records screened

• False-positive rate = 3.5% of industry average*

• 100m records screened daily (name matching only)

• False-positive rate = 10% of industry average*

Large Financial Services Company

Tier 1 Global Financial Services Company

*Using data sourced from UK FSA report into Financial services firms‟ approach to UK sanctions - April 2009


reserved.

Q&A





Documents

1 Copyright © 2011, Oracle and/or its affiliates. All rights€¦ · Including Duplicates and Missing Information. Product ID Manufa-cturer Description Product Type Power Voltage