Upload
boefbrak
View
217
Download
0
Embed Size (px)
Citation preview
8/6/2019 Real-Time Business Intelligence
1/35
Evolution In Information AccessReal-time Business Intelligence
8/6/2019 Real-Time Business Intelligence
2/35
Evolving to real-timedata availability
29%instantaneous
44%daily
24%weekly
3%monthly
Data: Gartner Survey of 540 organizations
2006 2006 2002 2002
55%daily
11%instantaneous
17%weekly
17%monthly
8/6/2019 Real-Time Business Intelligence
3/35
Real Time DataWarehouse DesignChallengesDr. Ralph KimballFounder Kimball Group
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
4/35
Definition of Real Time
Real Time is anythingthat is too fast for your current ETL
When you reach the performance limitations of your current ETL architecture, you are forced tore-architect your system.
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
5/35
Study the Alphabet Soup
EII Enterprise Information Integration Direct access to source data in warehouse framework
CTF Capture, Transform and Flow Database to database pipe
SOA Service Oriented Architecture XML documents with WSDL payloads inside SOAP envelopes
SETL Streaming ETL Record at a time transfer MBETL Micro Batch ETL
Batch transfer every few minutes or seconds
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
6/35
Real Time Requires More
But when you re-architect your system, you mustinclude a whole set of other new data warehouserequirements Operational Interaction
Extreme Integration Multi-modal data, especially text search Subtransactional data streams
ComplianceSome of these conflict
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
7/35
Operational Interaction
Most extreme latency demand Always need the current truthNeed write privileges with locking
Push, pull, and non-eventsGood application of EII Conflicts with integration and compliance
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
8/35
Extreme Integration
Local and global de-duplication in real time Need multi-valued back pointers to original IDsThe wider the integration,the more difficult to support real timeWorld is rapidly moving to SOA Service Oriented Architecture
Abstract individual system details Compose coarse grain services from fine grain XML data transfer but relational targets
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
9/35
Multi-modal data
Especially text search Google has rewritten expectations
for content and performanceAll of the web
Deliver search result in less than 1 secondCan expect hybrid combinations of SQL and text search on same databases Fact Tables for Text Document Searching(Kimball article archive: www.kimballgroup.com) old Red Brick pattern index
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
10/35
Subtransactional Data Streams
Enormous volumes, up to 10 TB/day threatenconventional ETL and RDBMS modelsData streams allow sequential tracking of individualcustomers or productsWe barely understand sequential behavior analysis Most queries today look at flow at a single point
Need to look at voyages through many points New design: Step dimension
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
11/35
Snippet from Modeling Class:Design of a Step Dimension
Design: Add a Step Dimension to a transaction fact table:
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
12/35
Compliance
Maintain the custody of data Prove later that data hasnt changed and
that you computed the result correctly Show how information has been used
Column-level encryptionHash-coded data sets Conflicts with operational interaction
Where is the system of record? Compliance webinar (April) on Informatica web site
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
13/35
How to Proceed(Building a RT Data Warehouse)
Decompose the problem before implementing! Business Needs Compliance Data Profiling
Security Data Integration Data Latency
Archiving and Lineage End User Delivery Interfaces Available IT and End User Support Skills
Legacy Licenses2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
14/35
How to Proceed
Choose an extensible architecture good at Simultaneous updating and querying Enormous data volumes, up to TB/day Multi-modal data search
Compliance support Hardware independence
Learn, learn, learn Sybase Education Services (www.sybase.com) Kimball University (www.kimballgroup.com)
2005 Kimball Group / Kimball University All rights reserved.
8/6/2019 Real-Time Business Intelligence
15/35
Evolution In Information AccessReal-time Business Intelligence
8/6/2019 Real-Time Business Intelligence
16/35
Real-timeData Integration for Real-timeBusiness IntelligenceDavid LyleVice President, Product StrategyInformatica Corporation
2005 Informatica Corporation
8/6/2019 Real-Time Business Intelligence
17/35
2005 Informatica Corporation
Real-time BI requires real-time DI
Real-time Business Intelligence requires theintegration of data in real-time
Kimball definition of real-time: Anything that is too fast for your current ETLYou need to re-architect your kitchen, but you still need a sink,oven, stove, fridge,
I f i l i DI
8/6/2019 Real-Time Business Intelligence
18/35
2005 Informatica Corporation
Informatica real-time DIto the rescue
Different appliances for your real-time kitchenEII virtual integration through query federation
Capture, transform and flow = PowerCenter Real-time optionSOA PowerCenter Connect for Web servicesPowerExchange Change Data Capture options
(Streaming ETL, Micro-batch ETL)PowerCenter Unstructured Data option
8/6/2019 Real-Time Business Intelligence
19/35
2005 Informatica Corporation
Capture, transform and flow
Always-on sessionsPowerCenter Connect for MSMQ, IBM MQSeries, Tibco,
webMethods, httpCan perform the extreme integration youre used to within thesame PowerCenter platform
8/6/2019 Real-Time Business Intelligence
20/35
2005 Informatica Corporation
Service-oriented architecture
New way of thinkingNew programming language, too
But SOA doesnt equal real-time data integrationSOA frameworks are enterprise stub-generators, leaving the dataintegration problem up to you!
PowerCenter Connect for Web servicesBuild your data integration logic as Web services, backed up bymetadata and change management
8/6/2019 Real-Time Business Intelligence
21/35
2005 Informatica Corporation
Change data capture
Approaches to dateAdd timestamps through reverse engineering
User exits that interrupt transaction streamsAdd triggers that impact application performanceFull file compares to determine deltas
Home-grown solutions technically challenging
all disruptive, all programming-centric!
8/6/2019 Real-Time Business Intelligence
22/35
2005 Informatica Corporation
Value
Simplify with one standard for all data types and latenciesExtend value of existinginvestments
Accelerate project lifecyclesReduce solution developmentand maintenance costs
Technology
Complex source accessPatented changed data captureLatency: real-time, change, batchNon-invasive, service-orientedarchitectureCodeless, visual SQL drivenEnterprise scalability
Informatica PowerExchange
Unlock complex data on demand
8/6/2019 Real-Time Business Intelligence
23/35
2005 Informatica Corporation
Real-time analytics
The reduction in data integration overhead may beits own justification for a real-time data warehouse
Less data to move, reduce the load window, keepup with increasing data volumes, smaller systemand disk sizes requiredIts a win, win, win situation!
And then if the business also has aBI requirement for real-time, thats obviously
the most important justification of all!
8/6/2019 Real-Time Business Intelligence
24/35
Evolution In Information AccessReal-time Business Intelligence
8/6/2019 Real-Time Business Intelligence
25/35
Business Intelligenceon DemandInformation at the right time,right place, right priceKapi Attawar Senior Director Sybase, Inc.
2005 Sybase, Inc.
8/6/2019 Real-Time Business Intelligence
26/35
Sybase solutions
2005 Sybase, Inc.
ON-DEMANDMOBILITY SERVICES
AvantGo
Mobile Business Services AvantGo Premier Hosted PortalHosted Emails
INFORMATIONANYWHERE
SQL AnywhereM-Business AnywhereTM
Mobile Link
Unwired Accelerator Unwired Orchestrator
Answers AnywhereTMMobile Sales FamilyPylonTM
AfariaMach DesktopTM
mFolioTM
RFID Enterprise/RFID AnywhereTM
HIGH-PERFORMANCEDATABASE
ASE
ENABLE BETTER DEVELOPMENTWorkSpace, PowerDesigner , PowerBuilder , PocketBuilder TM for client/server, Web and mobile
INFORMATION MANAGEMENT (DATACENTER) INFORMATION MOBILITY (DE
DATA
KNOWLEDGE
TAMING THEDATA EXPLOSION
Heterogeneous Data Services
Dynamic ArchiveMirror Activator TMDynamic ODS
Sybase IQ Auditing
Replication Server Real-Time Data Services
Enterprise InformationIntegration (EII)
Unstructured Data
8/6/2019 Real-Time Business Intelligence
27/35
Business problemCritical data in silos and in
different formsUsers need dynamic access todata in real time
Solution: Avaki EII
Cost-effective tool to developapplicationsSeamless access to disparatedata sources
Provides customers with virtualviews of: Multiple data sources Federated data access SOAIntegration and query capabilities
Business problemRapid growth of enterprise
information outside structureddatabasesThis unstructured data comesin a variety of formats emails,videos, PowerPoint, etc
Solution: ISDDProvides enhanced query andanalysis of both structured andunstructured dataAddresses both fixed andmobile sources
Recent acquisitions
2005 Sybase, Inc.
Just-in-time
8/6/2019 Real-Time Business Intelligence
28/35
Extract
DataWarehouse
Just in timeinformation architecture
OperationalData Store
ASE
Analytics
Business ObjectsCognos
Microstrategy
HyperionSPSSSAS
Crystal ReportsMicrosoft Excel
SAP BW
Applix
2005 Sybase, Inc.
Just-in-time
8/6/2019 Real-Time Business Intelligence
29/35
Transactions
Access
Avaki
Extract
DataWarehouse
Just in timeinformation architecture
OperationalData Store
ASE
Analytics
Business ObjectsCognos
Microstrategy
HyperionSPSSSAS
Crystal ReportsMicrosoft Excel
SAP BW
Applix
The business opportunityDemand for real-time business reportingand analyticsImprove service to business by movingaway from batch reporting
2005 Sybase, Inc.
Wh d I ?
8/6/2019 Real-Time Business Intelligence
30/35
What do I use?
Sybase Dynamic ODS with InformaticaOperational system overload and higher operational costs causedby data and user growthCut costs reporting from OLTP systems is very expensiveImprove performance transactions currently adversely affectonline reportingNeed both current and historical data
EII with Sybase AvakiOperational systems overload not the primary challengeOffload is not a good option too hard, data too dynamic, too muchfragmentation, mix of operational and OLAPReal-time accessibility with caching for performance and resourcemanagement
Ideal for a grid environmentMoving to an SOA architecture 2005 Sybase, Inc.
Real-time reporting
8/6/2019 Real-Time Business Intelligence
31/35
Real time reportingDynamic ODS with Informatica
ETL Tool (Informatica)
Harvest data in near real-timeSource database agnosticHigh-speed data harvestingDisaster recovery capability
Data on-demandwithout stressing theOLTP systemRisk-free approach,does not interfere withoperational systems
Dynamic ODS (Sybase)
Non-intrusive, risk freeData compression (up to 60%) for structuredand unstructured dataOptimized for fast-read access column-
based, not rowStandard SQL access to data
2005 Sybase, Inc.
Real-time reporting
8/6/2019 Real-Time Business Intelligence
32/35
ea t e epo t gEII with Avaki
EII platform (Avaki)
Real-time view of operational dataSource database agnosticHigh-speed data queries across distributedsystems
Configurable caching for improved performanceand reduced operational impact
Complex operationalenvironments
Managed approachto operationaldata access
Dynamic data transformationon-the-flyStandard SQL access to dataSupports updates and queries
Service Oriented Architeture
2005 Sybase, Inc.
DatabaseEnvironment
ETL Tool
Data Sources
Avaki Views
Streamlining mainframes
8/6/2019 Real-Time Business Intelligence
33/35
gand Teradata
Dynamic ODS customer case studiesPicknPay (South Africa) Replaced 20 node Teradata with 8 CPU Xeon environment
Major global credit card company (USA) Fraud detection investment of $4m = saving of tens of millions
Telefonica (Europe) Single view of customer - investment $4m = savings of $15m in first year Mainframe, Oracle and Teradata co-existence
HIRA (Korea) Enabled storage of five years of data on-line, improved query time from 4 days to 30 seconds Beat Teradata, DB2 and Oracle on performance and cost
Avaki customer case studies
Pfizer (USA) Daily sales dashboard real-time access to data from multiple data sources 600% improvement in analytic performance
Pacific Capital Bank (USA) Risk management single view of customer and risk scoring
Real-time access to data, transform and present in application-specific format
2005 Sybase, Inc.
Information at the right time,
8/6/2019 Real-Time Business Intelligence
34/35
g ,right place, right price
NeedsRelieve operational system overloadCut operational costsImprove transactional performanceProvide both current and historical data
Solution: Sybase Dynamic ODS and Informatica
NeedsOffload not a good optionAccess to operational system data in real-timeProvide a single virtual view of dataMergers and transitions
Solution: EII with Sybase Avaki
2005 Sybase, Inc.
8/6/2019 Real-Time Business Intelligence
35/35
Evolution In Information AccessReal-time Business Intelligence