Data Virtualization

Composite Data Integration

Peter Tran

April 15, 2008

Data Virtualization

Data Virtualization – The Unstoppable Force

The Virtualization TrendWhat is Data VirtualizationHow it WorksWhy VirtualizeWhen to Virtualize, When to ConsolidateData Virtualization and SOAGetting StartedQuestions and Answers

Next-Generation Virtualization

Disk 1 Disk 2 Disk 3 Disk 4

Application 1 Application 2 Server 1 Server 2

Data Warehouses

RDBMS Flat FilesPackaged Apps

Web Services

Storage Virtualization Abstracts logical storage from physical storage

Data Virtualization Abstracts data location and complexity

Abstracts logical apps & servers from physical apps & servers

Application / Server Virtualization

Common Thread – Real Value to Business & IT

Overcome (hide) ComplexityHardwareSoftware

Improve AgilityNew solutions Existing solutions

Reduce CostsOperatingNew development

Storage Virtualization

Data Virtualization

Application / Server Virtualization

Data Warehouses

Web Services

Data Virtualization - The New Frontier of Data Integration

ExistingData

ConstantChange

Siloed&

Complex

DataIntegrationChallenge

BusinessSolutions

ReportingDashboards Applications

Code within the Application? File Extracts / Database Replications?ETL with Warehouses, Marts, Stores?

Virtual Data Federation / EII?

DIOptions

But This Sounds A Lot Like…..

Enterprise Information Integration (EII)High Performance Query / Data Federation SOA Data Services Information As A Service (IaaS)…

Terms slightly different, but core concepts are the same.

Virtualize data silos Abstract away complexityFederate heterogeneous data

Customers

Orders

Invoices

Trades

Positions

Vendors

Employees

Data Warehouses

Web Services

Three Core Concepts At A Glance

ExistingData

Real-timeInformation

Siloed&

Complex

VirtualizeVirtualDataLayer

Abstract

BusinessSolutions

Federate

Data Virtualization – From an Analyst’s Perpsective

Challenge:“Enterprises are facing growing challenges in using disparate sources of data managed by different applications, including problems with integration, security, performance, availability, and quality.

Requirements:Business users need fast, real-time, and reliable information to make business decisions, while IT wants to lower costs, minimize complexity, and improve operational efficiency.

Solution: New technology is emerging that Forrester has dubbed ‘information fabric,’ defined as a virtualized data layer that integrates heterogeneous data and content repositories in real time.”

"Information Fabric 2.0: Enterprise Information Virtualization Gets Real" by Noel Yuhanna, Mike Gilpin

Project Level Data Virtualization – Global Pharmaceutical

Research Scientist Workbench

Situation and Requirements:Researchers require on-demand access from many sources to speed drug discovery projectsRapid applications development approachSOA compliance

Solution: • Composite combines disparate data for

Research Scientist WorkbenchResults:

50% reduction in development time for complex data integration projectsSOA-compliance, with data services in the format required by data usersData quality improved through rapid prototypingReduced cost of operational infrastructure

XML data

Packaged Apps

HTTP files

Historical Data

Virtual Data Mart

Research Scientists WorkbenchDrug Discovery

SOA to the Rescue, When Drug Discovery Needs Data Fast! Dan Eng, Pfizer, SOA World Magazine

Project Level Data Virtualization – Top 5 Investment Bank

Mortgage Loan Management

Situation and Requirements:10,000 Loans in process Management seeks visibility and controlOperations needs to complete funding stepsFive source systems, cannot replicate data

Solution:Virtual data mart accessing and delivering data via Web data servicesProvide real-time data to Management Portal and Workflow System

Results:Management get visibility on funding needs, exposure, operating metricsLoan officers process loans fasterSensitive data is protected

ManagementReporting

Virtual Data Mart

Loan ProcessingWorkflow Mgmt

LoanOrigination

RiskAnalysis

Loan Approval

LoanFunding

Enterprise Data Virtualization - Global Pharmaceutical

Situation and Requirements:More up-to-date information needed for

HR Operations ManagementSales Operations ManagementDrug Discovery

Share data across the companyStructured and semi-structured dataMultiple consuming front-ends

Solution: Virtual data layer using Composite Information ServerReusable views and data services for sourcing and delivering data

Results:Simplified data sharing across the companyBuild business applications faster to keep up with changing businessCost savings in reusing data services

SalesforceCompensation

GlobalPharma

Prescription

GeographicCore

Master

Call ClusterPlanningFinancial

Warehouse

HR OperationsManagement

Sales OperationsManagement

Drug Discovery

Virtual Data Layer

Enterprise Data Virtualization - Top 5 Investment Bank

Situation & Requirements:Up to the minute information needed for Risk management, trade order management, and Debt/equity market researchShare data across the bankSimplified access to master & reference data stored in Teradata and other sourcesServe data to .NET & other front ends

Solution: Implement data virtualization layer using Composite Information Server Virtualize data for delivery in format needed by consuming applications25 Applications, 200+ Sources

Results:Simplified data sharing across bankFaster access to reference & transaction dataCost savings in reusing data services

Virtual Data Layer

Sybase -Product data

SQL Server -Coverage

Holdings &Offerings

Teradata -Cache

Trade OrderManagement

RiskManagement

Market Research

Data Warehouses

Web Services

Design Time - Easy to Use Modeling & Code Generation

ExistingData

DesignTime

Functions

VirtualData Layer

BusinessSolutions

• Drag and Drop Designer• Metadata Integration & Mgmt • Standard Drivers • Pre-built Enterprise Apps

Data Services

• Automated CodeGeneration

• XML/SQLTransformations

• Scripting Optional

Data Warehouses

Web Services

Run Time - High Performance Query & Reliable Delivery

ExistingData

Run TimeFunctions

VirtualData Layer

BusinessSolutions

• Access / Connectivity• Query Optimization• Federation• Abstraction• Delivery via Service or View

• Data Security• Caching• Clustering• Flexible Management• Technology Independent

Virtualize Data To Increase Agility

Deliver Solutions2-4x Faster

Time AdvantagesConfigure, not codeNo need to build & test martQuicker iterationsLeverage abstractionsReuse prior work

Virtual DataMarts

File Extracts

Time toSolution

The VirtualDifference

Virtualize Data To Reduce Costs

Reduce DI Costs by 25-50%

Marts, Extracts,

Other Costs

VirtualData Integration

The VirtualDifference

Cost AdvantagesFewer data martsFewer data extractsBetter data controlBetter leverage of existing resourcesMore flexible architecture

Virtualize, Consolidate, Replicate?

Need to considerPeople…..Skills, ICC, Politics…Process….SDLC, Architecture Review, PPM…Technology….Virtualize, Consolidate, Replicate, or maybe even a Combination…

Factor-based AnalysisBusiness Consideration…Data Source Considerations…Data Consumer Considerations

A Simple Tool to Help You Decide

Combined Approaches Sometimes Provide Best of Both Worlds

European TelecomNear real-time sales & customer management reporting in high transaction volume environment

Consolidation / ETL Reporting data mart refreshed hourlyReduce reporting load on operational (Siebel CRM) system

Data Virtualization / EIIOne face to the reporting portal system (Cognos)Combine real-time and historical data

Data Virtualization with EII

Data Consolidationwith ETL

CRMApplications

ReportingData Mart

NetworkApplications

FinancialWarehouse

Reporting Portal

Business Solutions

Transaction Services

App Server

1) Data services stand alone2) Transaction services stand

alone3) Data services integrated

with transaction services4) Data services with

messaging using an ESB5) Data Services with a BPM6) Data services WSDL stored

in a UDDI registryData Services (Info Server)Data Services

Info ServerDATADATA

DATADATA

DATADATA DATA

SOA Components Working Together

UDDIRegistry

“Over 60% of SOA services will beData Services”

Source: IDC

“Over 60% of SOA services will beData Services”

Source: IDC Data Warehouses

Web Services

ESB – Messaging/Comm System

Next Steps - Guidance

Keep an open mind Data Virtualization is a NOW a proven option with significant time and cost saving benefits It’s not always either/or…sometimes it’s both

Leverage what you already knowModeling and SQL are common denominatorsIntegration Competency Centers

Get startedPick a project and a SWAT teamSelf-fund expansion with savings

Take advantage of enterprise synergiesBusiness Intelligence, Portals, SOA, CDI-MDM…Success of the “Other” virtualizations

Thank You

Peter TranComposite Software

peter@compositesw.com

Data Virtualization

Technology

Data Virtualization – Flexible Technology for the Agile ... Virtualization – Flexible Technology for the Agile Enterprise ... Data Virtualization – Flexible Technology for the

Composite Data Virtualization and Data Service Solutions

TIBCO® Data Virtualization

Data Center Virtualization

Data Virtualization in the Cloud: Accelerating Data Virtualization Adoption

Data Virtualization or ETL - denodo.com for applications performing data mining or historical ... updates to the data. Data virtualization, ... Comparison of ETL/EDW vs. Data Virtualization

Leveraging Data Virtualization - Axians · data sources – readily available to your data analysts and data scientists ü Data virtualization shortens the ‘data wrangling’ phases

JBoss Enterprise Data Services (Data Virtualization)

Composite Data Virtualization

Data Virtualization and Visualization

Oracle Consulting Virtualization Services Data Sheet DATA SHEET 3 Oracle Consulting Virtualization Architecture Service The Virtualization Architecture Service provides architecture,

IBM Data Virtualization Manager for z/OS: Data ......2000/03/02 · The Data Virtualization Manager server messages have a format that is consistent with the Data Virtualization Manager

Lesson 10 Virtualization & Data Centric Network · Lesson 10 Virtualization & Data Centric Network ! Objectives : Network function, system and link virtualization are new techniques

Creating an Agile Data Integration Platform using Data Virtualization€¦ · Data Virtualization in a Nutshell The Advantages of Data Virtualization Data Virtualization as Basis

Data Virtualization Usage Patterns

Data Center Virtualization @ Cisco

WHITEPAPER Why Data Virtualization is an …...virtualization. Data virtualization doesn’t solely have to exist within a data warehouse. Any time data is flowing from point A to

DATA VIRTUALIZATION - CloudFectiv · DATA VIRTUALIZATION Data virtualization is synonymous with information agility – With this technology organization’s are able to provide a

Data Management Cisco Data Virtualization · Cloud Data Sources Big Data / IOE ... Cisco Data Virtualization is agile data integration software ... Oracle EBS, Salesforce.com,

Data Virtualization leads the CIO agenda - Home - InfoRoad...Digital Transformation for legacy modernization & data marketplaces Data Virtualization “You Need A Data Virtualization