Upload
mark-rittman
View
117
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Presentation by Stewart Bryson, Rittman Mead, on real-time data warehousing techniques using the Oracle Database and Oracle Business Intelligence Enterprise Edition. As delivered at Collaborate'11, Orlando, April 2011.
Citation preview
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Stewart Bryson, Managing Director, Rittman Mead AmericaCollaborate, Orlando, April 2011
Real-Time DW with OBIEE and Oracle Database
1Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Who am I?
2Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
• Stewart Bryson, Managing Director, Rittman Mead America
• Oracle BI/DW Architect and Delivery Specialist
• Oracle Database and BI Stack since 1996
• Regular Community Speaker
• Writer for Rittman Mead Blog:http://www.rittmanmead.com/blog
• Developer of Transcend Framework
• Email : [email protected]
• Twitter : @stewartbryson
3Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Who is Rittman Mead?
4Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
• Oracle BI&DW Experts
• Consulting, Training and Support
• Global Company, Offices in the US, UK and India
• Clients in the UK, USA, Europe, Middle-East
• Voted UKOUG BI Partner of the Year 2008
• 16 Sessions at Oracle Open World 2010
• Twitter : @rittmanmead
5Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Rittman Mead BI Forum, May 2011
6Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Registrations open for the Rittman Mead BI Forum, May 2011
• Brighton, UK, May 18-20th 2011• Atlanta, GA, May 25-27th 2011• Covers OBIEE, DW, EPM• Guest speakers including:‣ Jeff McQuigg‣ Kevin McGinley‣ Jean-Pierre Dijcks‣ Mike Durran‣ Shyam Varan Nath‣ Philippe Lions‣ Emiel van Bockel
• OBIEE 11g Masterclass withMark Rittman and Tony Heljula
• Full details at http://www.rittmanmead.com/biforum2011
7Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
What will this session answer?
8Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
• What do I mean by “real-time data warehousing?”‣ Zero latency, or just reduced latency?‣ Do I really mean “right-time data warehousing?”
• What can I use to reduce latency?‣ What does OBIEE offer?‣ What does the Oracle Database (and OWB)
offer?• Can I blend these different offerings?‣ Of course you can!‣ It’s the name of the presentation
9Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
What do I mean by “real-time”?
10Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
• Traditional data warehouses‣ Daily to weekly batch-load cycles‣ Key-metrics are always delayed‣ Information may not be “actionable”
• Real-time data warehouses‣ Up to the minute or hour reporting‣ Key-metrics are moving‣ Information is actionable
11Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
An interval will always exist between the occurrence of a measurable event and our ability to process that event as a “reportable fact”
12Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
An interval will always exist between the occurrence of a measurable event and our ability to process that event as a “reportable fact”
!"##$%!&'"()!"#"$%&'(##%)*+)%'(,-*./")%0*1"%#(%2"%)3$"%(4%#0"%.""5%2"4($"%#0"+%/.)/)#%(.%$"*67#/,"%23)/."))/.#"66/8".'"9
!"#$%&#'()*#&)%+,-./)#$(/01-.23#4(&#%-#+)%*-#5#6(0)#-6)#)23.2)#/%2%3)/)2-7#-&%$-.(2#$(2-&(+#%2'#89:#*"*-)/*#%&)#&)%+#-./);<=
!"#/(>.+)#06(2)#.*#&)%+#-./)7#)?)2#*(/)#(@#/"#6(/)#)2-)&-%.2/)2-#3%'3)-*#%&)#&)%+#-./)#A#>1-#B6%-#%>(1-#>1*.2)**
.2-)++.3)2$)#*"*-)/*C
52#-6)#)%&+"#'%"*#(@#95#4(&#B%*#.-#$%++)'#')$.*.(2#*100(&-#>%$D#-6)2C<#&)0(&-.23#'%-%#@(&#-6)#$1&&)2-#/(2-6#B%*#%2#%$6.)?)/)2-=
8*#-)$62.E1)*#%2'#-)$62(+(3.)*#)?(+?)'7#$(/0%2.)*#/(?)'#-(#&)0(&-#(2#-6)#0&)?.(1*#'%"#>1-#B)&)#*-.++#&)+.%2-#(2#%2#(1-,(@,
6(1&*#>%-$6#0&($)**#-(#/(?)#'%-%#@&(/#-6)#*(1&$)#*"*-)/*#%2'#*-(&)#%2'#%33&)3%-)#.-#.2#-6).&#'%-%#B%&)6(1*)*#%2'#'%-%
/%&-*=
91-#B.-6#FG,6(1&#>1*.2)**#'%"*#%2'#-6)#2))'#-(#&)0(&-#%$&(**#/1+-.0+)#-./),H(2)*7#-6)#-&%'.-.(2%+#>%-$6#B.2'(B#.*#>).23
*E1))H)'#.2-(#2)%(2,)I.*-)2$)=#J)B#'%-%#)I-&%$-.(2#%2'#+(%'#0%&%'.3/*#6%?)#%+*(#>))2#')?)+(0)'#-(#-&.$D+),@))'#&)0(&-.23
*"*-)/*=#91-#%&)#-6)*)#&)%++"#&)%+#-./)#A#%2'#'()*#.-#&)%++"#/%--)&C
52#&)%+.-"7#-6)&)#.*#%+B%"*#3(.23#-(#>)#%#')3&))#(@#+%-)2$"#>)-B))2#%2#)?)2-#6%00)2.23#%2'#.-#>).23#&)$(&')'#(2#%#-&%2*%$-.(2%+
*"*-)/7#0&(0%3%-)'#-(#-6)#&)0(&-.23#*"*-)/7#+(%')'7#%2'#@.2%++"#%33&)3%-)'#(2#%#'%-%#B%&)6(1*)#>)@(&)#>).23#&)%'"#-(#E1)&"=
8''#-(#-6%-#-6)#')+%"#>)-B))2#-6)#1*)&#.**1.23#%#E1)&"#-6%-#%$$)**)*#-6.*#2)B#2133)-#(@#.2@(&/%-.(2#%2'#-6)#&)*1+-#>).23
&)-1&2)'#-(#-6)#1*)&7#%2'#"(1#@.2'#%#'.*$)&2.>+)#+%3#>)-B))2#)?)2-#%2'#(>*)&?%-.(2=
K&1)7#-6)&)#%&)#-6.23*#"(1#$%2#'(#-(#/.-.3%-)#-6.*#+%3#>1-#-6)&)#B.++#%+B%"*#>)#%#$(10+)#(@#*-)0*#.2#-6)#0&($)**#B6)&)#&)'1$.23
(2)#-./)#0)&.('#.2$&)%*)*#%2(-6)&#A#@(&#)I%/0+)7#"(1#$%2#*6(&-)2#&)0(&-#)I)$1-.(2#-./)#>"#%33&)3%-.23#-6)#'%-%#-(#>)--)&
%2*B)&#-6)#E1)&"7#>1-#-6.*#*-)0#%''*#/(&)#-./)#-(#-6)#'%-%#+(%'#%2'#%33&)3%-)#06%*)=
L1&-6)&/(&)7#'.@@)&)2-#-"0)*#(@#.2$(/.23#'%-%#$%2#6%?)#'.@@)&.23#+%-)2$.)*=#L(&#)I%/0+)7#%#2)B#-&%2*%$-.(2#/%"#6%?)#%#*6(&-)&
+%-)2$"#-6%2#%#$6%23)#-(#%#&)3.(2%+#3&(10.23#(@#*-(&)*#-6%-#/%"#2))'#-(#>)#0&(0%3%-)'#-6&(136#/%2"#+%")&*#(@#6.*-(&.$%+
%33&)3%-.(2#-%>+)*#4*))#L.31&)#M<=
K6)#'.?.*.(2#>)-B))2#'%-%#%$E1.*.-.(2#%2'#'%-%#0&)*)2-%-.(2#0&)*)2-*#%#D)"#E1)*-.(2#-6%-#%2"#$(/0%2"#2))'*#-(#%2*B)&#B6)2
$(2-)/0+%-.23#/(?.23#-(#%#&)%+,-./)#95#*"*-)/N#%&)#"(1#O1*-#.2-)&)*-)'#.2#0&(?.'.237#%*#&%0.'+"#%*#0(**.>+)7#.2@(&/%-.(2#>%*)'
(2#%+&)%'"#%$E1.&)'#'%-%7#(&#'(#"(1#%+*(#2))'#-(#&)0(&-#(2#2)B+"#%'')'#@%$-1%+#.2@(&/%-.(2C
L(&#)I%/0+)7#%#B(&D)&#.2#%#/(>.+)#06(2)#$(/0%2"P*#$%++
$)2-&)#/%"#+.D)#-(#*))#.2@(&/%-.(2#%>(1-#Q6(B#3(('#%
$1*-(/)&#.*P#%2'#-6)#+.D)+.6(('#-6%-#*(/)(2)#B.-6.2#-6%-
$1*-(/)&P*#')/(3&%06.$#B(1+'#*B.-$6#-(#%2(-6)&#
*100+.)&=
K6.*#.2@(&/%-.(2#2))'*#-(#>)#0&)*)2-)'#E1.$D+"#4B6.+*-#-6)"
%&)#*0)%D.23#-(#-6)/<#%2'#0&)@)&%>+"#B.-6.2#-6)#$(2-)I-#(@#-6)
+.2),(@,>1*.2)**#%00+.$%-.(2#-6%-#/%2%3)*#-6)#$1*-(/)&
.2-)&%$-.(2=#R(B)?)&#.-#.*#12+.D)+"#-6%-#%2"#(@#-6)#/)-&.$*#1*)'
>"#-6)#$%++#$)2-&)#%3)2-#@(&#$1*-(/)�&(@.-%>.+.-"#%2'
0&)'.$-)'#$61&2#B(1+'#>)#.2@+1)2$)'#>"#)?)2-*#(@#-6)#+%*-#@)B
6(1&*=
S(2?)&*)+"7#%#*"*-)/#-6%-#.*#1*.23#)/>)'')'#95#@12$-.(2%+.-"
-(#%+)&-#-(#0(-)2-.%+#@&%1'#4*-($D#/%&D)-*#%2'#$&)'.-#$%&'
$(/0%2.)*#%&)#0(**.>+)#)I%/0+)#1*)&*<#/%"#B)++#2))'#-(
D2(B#%>(1-#10,-(,-6),/(/)2-#%$-.?.-"=
***+,-./0.12345,416,+537
FIGURE 1: BI latency
Recording Propagation Aggregation Query
Event Report
Real-time query latency
12Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Business Case
13Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
• We want to be able to segment our reporting between “real-time” needs and static reports
‣ Most users will want real-time numbers delivered with dashboards
‣ Some users (financial-minded) don’t want numbers to change each time they run a report
• Use the standard Oracle BI Stack
‣ Oracle Database Enterprise Edition
‣ Oracle Business Intelligence Enterprise Edition
14Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Latency versus Query Performance
15Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Latency versus Query Performance
OLTP Reporting
Federated OLTP/EDW Reporting
EDW with Realtime Component
Standard EDW
15Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Latency versus Query Performance
OLTP Reporting
Federated OLTP/EDW Reporting
EDW with Realtime Component
Standard EDW
Latency
Query Performance
15Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Latency versus Query Performance
OLTP Reporting
Federated OLTP/EDW Reporting
EDW with Realtime Component
Standard EDW
Latency
Query Performance
15Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Oracle Next-Generation Reference Architecture
16Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Requirement: Identifying and Capturing Change
17Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Requirement: Change Data Capture
18Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Requirement: Change Data Capture
Identify Change
• Oracle CDC• OWB Trickle-feed
mappings• Trigger-based
capture• Recording
modification dates• MINUS Command
18Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Requirement: Change Data Capture
Identify Change
• Oracle CDC• OWB Trickle-feed
mappings• Trigger-based
capture• Recording
modification dates• MINUS Command
Propagate Change
• Oracle CDC• OWB Trickle-feed
mappings• Message Queues• Views over
database links• File movements
18Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Requirement: Change Data Capture
Identify Change
• Oracle CDC• OWB Trickle-feed
mappings• Trigger-based
capture• Recording
modification dates• MINUS Command
Propagate Change
• Oracle CDC• OWB Trickle-feed
mappings• Message Queues• Views over
database links• File movements
Apply Change to Target
• OWB Mappings and Process Flows
• Other ETL Tools• Custom ETL code• Custom transaction
code
18Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Requirement: Change Data Capture
Identify Change
• Oracle CDC• OWB Trickle-feed
mappings• Trigger-based
capture• Recording
modification dates• MINUS Command
Propagate Change
• Oracle CDC• OWB Trickle-feed
mappings• Message Queues• Views over
database links• File movements
Apply Change to Target
• OWB Mappings and Process Flows
• Other ETL Tools• Custom ETL code• Custom transaction
code
18Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Oracle Asynchronous Change Data Capture
19Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Oracle Asynchronous Change Data Capture
Hot LogOnline redo logs mined locallyChange tables are updated and stored locally
Hot Log DistributedOnline redo logs are mined locallyChange tables are updated and stored remotely
Autolog OnlineOnline redo logs are mined remotelyChange tables are updated and stored remotely
Autolog ArchiveArchived redo logs are mined remotelyChange tables are updated and stored remotely
19Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Oracle Asynchronous Change Data Capture
Hot LogOnline redo logs mined locallyChange tables are updated and stored locally
Latency
Hot Log DistributedOnline redo logs are mined locallyChange tables are updated and stored remotely
Autolog OnlineOnline redo logs are mined remotelyChange tables are updated and stored remotely
Autolog ArchiveArchived redo logs are mined remotelyChange tables are updated and stored remotely
19Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Requirement: Change Data Capture
Identify Change
• Oracle CDC• OWB Trickle-feed
mappings• Trigger-based
capture• Recording
modification dates• MINUS Command
Propagate Change
• Oracle CDC• OWB Trickle-feed
mappings• Message Queues• Views over
database links• File movements
Apply Change to Target
• OWB Mappings and Process Flows
• Other ETL Tools• Custom ETL code• Custom transaction
code
20Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Requirement: Change Data Capture
Identify Change
• Oracle CDC• OWB Trickle-feed
mappings• Trigger-based
capture• Recording
modification dates• MINUS Command
Propagate Change
• Oracle CDC• OWB Trickle-feed
mappings• Message Queues• Views over
database links• File movements
Apply Change to Target
• OWB Mappings and Process Flows
• Other ETL Tools• Custom ETL code• Custom transaction
code
20Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Requirement: Change Data Capture
Identify Change
Propagate Change
Apply Change to Target
21Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Requirement: Change Data Capture
Identify Change
Propagate Change
Apply Change to Target
•Trickle-feed mappings allow the development of Streams Capture and Apply processes as continuously running mappings.
21Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Federated OLTP/EDW Reporting
22Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
• Layering in OLTP data to provide the real-time component
• General approach is to use a BI tool such as OBIEE
• No real database or modeling involvement
23Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Classic Batch Load Star Schema: Process Flow
24Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Classic Batch Load Star Schema: Fact Table Load
25Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Federation with OBIEE
26Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
OBIEE Metadata and Data Layers
27Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
OBIEE Metadata and Data Layers
Relational, OLAP, File and Application Data Sources
• Physical Data Sources, suchas relational databases,OLAP Cubes and applicationsource adapters
27Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
OBIEE Metadata and Data Layers
Relational, OLAP, File and Application Data Sources
• A Semantic Model made upof Metrics, dimensions, hierarchies and calculations, derived from...
• Physical Data Sources, suchas relational databases,OLAP Cubes and applicationsource adapters
Enterprise Semantic Model
27Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
OBIEE Metadata and Data Layers
• The Web Catalog containsdefinitions of reports, alertsdashboards, KPIs, based on...
Relational, OLAP, File and Application Data Sources
• A Semantic Model made upof Metrics, dimensions, hierarchies and calculations, derived from...
• Physical Data Sources, suchas relational databases,OLAP Cubes and applicationsource adapters
Web Catalog
Enterprise Semantic Model
27Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Flow of Data Through the Three-Layer Semantic Model
28Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Flow of Data Through the Three-Layer Semantic Model
Simplification of the Data ModelIntegration of Disparate DataSources
Addition of Business Logic and Calculations
28Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
29Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
29Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
29Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
29Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
30Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
30Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
30Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
31Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
31Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
31Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
31Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
32Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Map to Physical Models
32Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Respect the Physical Model
33Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Respect the Physical Model
33Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Logical Table Sources Respect the Physical Model
33Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
WITH SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2, T43901.CUST_LAST_NAME as c3, T43971.SAL_AMT as c4from GCBC_CRM.CUSTOMERS T43901, GCBC_POS.POS_TRANS T43971, GCBC_POS.POS_TRANS_HEADER T43978where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE('2010-09-18 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T43978.TRANS_DATE ) union allselect T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T44105.AMOUNT as c4from GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )),SAWITH1 AS (select sum(D3.c4) as c1, D3.c2 as c2, D3.c3 as c3from SAWITH0 D3group by D3.c2, D3.c3)select distinct 0 as c1, D2.c2 as c2, D2.c3 as c3, D2.c1 as c4from SAWITH1 D2
34Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
WITH SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2, T43901.CUST_LAST_NAME as c3, T43971.SAL_AMT as c4from GCBC_CRM.CUSTOMERS T43901, GCBC_POS.POS_TRANS T43971, GCBC_POS.POS_TRANS_HEADER T43978where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE('2010-09-18 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T43978.TRANS_DATE ) union allselect T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T44105.AMOUNT as c4from GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )),SAWITH1 AS (select sum(D3.c4) as c1, D3.c2 as c2, D3.c3 as c3from SAWITH0 D3group by D3.c2, D3.c3)select distinct 0 as c1, D2.c2 as c2, D2.c3 as c3, D2.c1 as c4from SAWITH1 D2
34Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
WITH SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2, T43901.CUST_LAST_NAME as c3, T43971.SAL_AMT as c4from GCBC_CRM.CUSTOMERS T43901, GCBC_POS.POS_TRANS T43971, GCBC_POS.POS_TRANS_HEADER T43978where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE('2010-09-18 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T43978.TRANS_DATE ) union allselect T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T44105.AMOUNT as c4from GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )),SAWITH1 AS (select sum(D3.c4) as c1, D3.c2 as c2, D3.c3 as c3from SAWITH0 D3group by D3.c2, D3.c3)select distinct 0 as c1, D2.c2 as c2, D2.c3 as c3, D2.c1 as c4from SAWITH1 D2
34Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
WITH SAWITH0 AS (select sum(T44105.AMOUNT) as c1, T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T45416.CALENDAR_YEAR as c4from GCBC_EDW.DATE_DIM T45416, GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') ) group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c1 as c5from SAWITH0 D1
35Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
WITH SAWITH0 AS (select sum(T44105.AMOUNT) as c1, T44042.CUSTOMER_FIRST_NAME as c2, T44042.CUSTOMER_LAST_NAME as c3, T45416.CALENDAR_YEAR as c4from GCBC_EDW.DATE_DIM T45416, GCBC_EDW.CUSTOMER_DIM T44042, GCBC_EDW.SALES_FACT T44105where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') ) group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)select distinct 0 as c1, D1.c2 as c2, D1.c3 as c3, D1.c4 as c4, D1.c1 as c5from SAWITH0 D1
• The BI Server understands from the Fragmentation content how to “optimize” the query
• Only one LTS is needed to provide the answer
35Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Fragmentation with OLTP and EDW
Demonstration
36Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
• Strengths
‣ Low latency
‣ No complex modeling or ETL
‣ Incorporated into any standard dimensional EDW
• Weaknesses
‣ Performance
‣ Difficult to conform multiple systems
37Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
EDW with a Real-Time Component
38Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
• Modeling complimentary real-time structures in an EDW
• Use functionality of the Oracle Database and OWB
• Complex ETL and Database interaction
39Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
The Real-Time Partition
Table Partition
Table Partition
Table Partition
Fact Table
40Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
The Real-Time Partition
Table Partition
Table Partition
Table Partition
Fact Table
Physical Table
Fact Real-Time Partition
40Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
The Real-Time Partition
Table Partition
Table Partition
Table Partition
Fact Table
Physical Table
Fact Real-Time Partition
40Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
The Real-Time Partition
Physical Table
• All ETL processing of fact records are directed at the Real-Time Partition using Micro-Batch
• No indexes, small data set• Partition Exchange at the end of the day
41Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
The Real-Time Partition
Physical Table
• All ETL processing of fact records are directed at the Real-Time Partition using Micro-Batch
• No indexes, small data set• Partition Exchange at the end of the day
41Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
The Real-Time Partition
Table Partition
Table Partition
Table Partition
Physical Table
• All ETL processing of fact records are directed at the Real-Time Partition using Micro-Batch
• No indexes, small data set• Partition Exchange at the end of the day
41Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
The Real-Time Partition
Table Partition
Table Partition
Table Partition
Physical TableTable Partition
• All ETL processing of fact records are directed at the Real-Time Partition using Micro-Batch
• No indexes, small data set• Partition Exchange at the end of the day
41Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Early-Arriving Facts
42Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
• Grab the most recent dimension record when that entity already exists
• Use a default “Not Known Yet” record when they don’t
• Prior to the partition exchange, we run our late-arriving dimension loads
• We update the Real-Time Partition with the correct surrogate keys‣ OWB generates an efficient MERGE‣ Small data sets in the Real-Time Partition
What if Type 2 dimension changes haven’t arrived yet?
43Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Real-Time Integration: Fact Table RT Load
44Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Real-Time Integration: Fact Table RT Load
44Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Real-Time Integration: Correct Dimensions
45Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Real-Time Integration: “Closing the Books” on the Day
46Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Plugging it together with OBIEE (Non-Fragmentation)
• Using standard database functionality (UNION ALL)
• Could be done with a view• Using Surrogate Keys in the
RT Partition• OBIEE doesn’t have to
manage different LTS’s
47Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Plugging it together with OBIEE (Non-Fragmentation)
• Using standard database functionality (UNION ALL)
• Could be done with a view• Using Surrogate Keys in the
RT Partition• OBIEE doesn’t have to
manage different LTS’s
47Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Plugging it together with OBIEE (Non-Fragmentation)
• Using standard database functionality (UNION ALL)
• Could be done with a view• Using Surrogate Keys in the
RT Partition• OBIEE doesn’t have to
manage different LTS’s
47Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
EDW with a Real-Time Component
Demonstration
48Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Summary
49Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
• Latency versus performance and scalability
• Latency and performance versus degree of difficulty
• Different variations on the two test cases
50Thursday, 21 April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : [email protected] W: www.rittmanmead.com
Stewart Bryson, Managing Director, Rittman Mead AmericaCollaborate, Orlando, April 2011
Real-Time DW with OBIEE and Oracle Database
51Thursday, 21 April 2011