Upload
dayna-hoover
View
218
Download
0
Embed Size (px)
Citation preview
Top Five Ways Swiss Mobiliar Improved Business Value with Oracle
Database In-Memory
Paolo KrethHead of Data Management
Thomas BaumannHead of IT Performance Management
1
In-Memory Making Headlines at Swiss Mobiliar New business insights due to real-time analytics
ANALYTICS ON OLTP DATAREPORTS AUF OLTP DATEN
IN 9 VON 10 FÄLLEN KEIN SQL QUERY TUNING MEHRIN 9 VON 10 FÄLLEN KEIN SQL QUERY TUNING MEHR
3
In-Memory Making Headlines at Swiss Mobiliar Massively reduced tuning efforts for SQL query texts
9 OUT OF 10 QUERIES DON‘T NEED SQL TUNINGREPORTS AUF OLTP DATEN
REDUCED ETL FLOW
In-Memory Making Headlines at Swiss Mobiliar A paradigm change: “Bring Analytics to the Data“
Verzicht auf ETL
4
5
Agenda
Swiss Mobiliar in a Nutshell Analytics on OLTP data – where we are today Analytics on OLTP data – PoC with Database In-Memory PoC Database In-Memory Results Further Experiences and Next Steps with Database In-Memory at Swiss Mobiliar Summary
legal form of a cooperative association (mutual company).
Switzerland’s number one insurer for household contents, business and pure risk life insurance.
close to customers throughout the country thanks to around 80 general agencies at 160 locations.
over 1.7 million insured persons or firms.
over 4,400 employees and 325 trainees.
6
Swiss Mobiliar Switzerland‘s most personal insurer
13x continuously 2003-2015
7
Overall Non-Life Insurance Market Growth in SwitzerlandClose to 2/3 of Market Growth to Swiss Mobiliar in 2014
Growth Mobiliar Market Growth
in Mio CHF. Source: Schweizerischer Versicherungsverband
8
Systems: iOS, Linux, Windows and z/OS 5000 Notebooks, 1800 iPhones, 300 iPads DBMS: Oracle, DB2, IMS, MS SQL Server, Neo4j Standard-SW (ArcGIS, SAP, BO, Siebel, COR Life etc.) Many home-grown applications
Swiss Mobiliar‘s IT Misson Statement: Deploy and Run innovative technologies for efficient business processes
9
The SpeakerThomas Baumann
9
Born in 1963 MSc. at Swiss Federal Institute of Technology, Zurich
Computer Sciences and Statistics Today, this mix is called Big Data
Has been focusing on Database Management Systems and Performance since 1992 Speaker at numerous international conferences „Performance Minister“ at SwissMobiliar
dedicated to performance – since 1963
„dedicated to performance – since 1963“ also produces this search result:
10
Agenda
Swiss Mobiliar in a Nutshell Analytics on OLTP data – where we are today Analytics on OLTP data – PoC with Database In-Memory PoC Database In-Memory Results Further Experiences and Next Steps with Database In-Memory at Swiss Mobiliar Summary
Analytics on OLTP Data at Swiss MobiliarArchitecture 2014
OLTP
Decision Support
Business Intelligence
Data Mart
Data Warehouse
Cross Information Systems
CoreInformation
Systems
Access Information Systems
Analytical (OLAP)
Operational(OLTP)
OLTP Analytics Scope
Analytics on OLTP data at Swiss MobiliarArchitecture 2020
OLTP
Decision Support
Business Intelligence
Data Mart
Data Warehouse
Cross Information Systems
CoreInformation
Systems
Access Information Systems
Analytical (OLAP)
Operational(OLTP)
OLTP Analytics Scope
13
Design Patterns
Column oriented data storage Data replication close to real time based on
log records, not transactionally consistent Query Re-Routing decided by optimizer,
transparent for application No need for indexes Very high compression rate
Architecture 2014IDAA (IBM DB2 Analytics Accelerator) Value Delivery
DB2 zOS(“Mainframe“)
IBM Netezza
SQL Query(to DB2)
Result Set
Automatic query re-routing of search-intensive queries1) to data copy at Netezza appliance
1) For applications which don‘t require transactionally consistent data, and can accept data delayed by a few minutes
Similar design patterns for Oracle Database In-Memory Are there similar results as well?
Major Results Increased OLAP query performance
100 times faster in average
Faster inserts on DB2 and higher scalability
Due to elimination of most indexes
Short timeframe between data ingress and analysis
30% REDUCTION OF MAINFRAME CPU
More Real-Time Analytics Headlines at Swiss Mobiliar Reduced CPU consumption during peak time
30% REDUKTION MAINFRAME CPU RESSOURCEN
14
IDAA in more detail(or what is different to Oracle Database In-Memory)
Massive Parallel Processing Architecture 32 nodes on 2 blades at Swiss Mobiliar Max Capacity 1024 nodes (8 racks)
Data Processed Locally FPGA (HW encoded logical arrays):
Decompression, Projection, Predicate Application all done locally within FPGA
Joins processed locally if possible, data redistribution among nodes otherwise
Data clustering within node to support skips for data scanning (zone maps)
Tables completely replicated (all rows/columns)
Tuning Knobs:
Data distribution among nodes Data clustering within nodesBoth are independent of DB2
partitioning and clustering
15
First Business Applications Ad-hoc reports from business end users Improved end-of-month processing Log analysis based on DB2 tables for access pattern analytics Improved ETL flow
Streamline Mainframe for OLTP Eliminating indexes used for analytics only Eliminate MQT and other auxiliary structures for analytics Reduced demand for reorg More efficient inserts
Step-by-Step IDAA Usage12 month from installation to broad usage
12 months
New Business Functions
We expect a similar timeline for Oracle Database In-Memory (see later)
16
New Business Reports (1 of 2)How Swiss Mobiliar‘s CRM system is used?
HQ
6 6.5 7 7.5 8 8.5 9 9.5 1045
45.5
46
46.5
47
47.5
48
“The more distant from HQ, the less they care about instructions“
How was that this morning(CEO visit at Hochdorf agency)?
report produced with 3.8sec response time
The larger the surface of the ball, the more the agency is following HQ‘s instructions on how to use Swiss Mobiliar‘s CRM system.
17
Owner of which car makes do also have many other contracts at Swiss Mobiliar?
New Business Reports (2 of 2) Cross Selling
with partner_MFZ (PNR, Make) as (SELECT DISTINCT PAR.C97251 AS PNR, MFZ.D63538 as MAKE FROM DB2PVIEW.VGESGPA2 PAR, DB2PVIEW.VALL0800 VER, DB2PVIEW.VPA08091 VER_OBJ,
DB2PVIEW.VOBMFZP1 MFZ WHERE PAR.C95826 = VER.C95826 AND VER.C95826=VER_OBJ.C95826 AND VER_OBJ.C95836_1=MFZ.C95836
AND VER_OBJ.C95862=MFZ.C95862 AND PAR.C97150 = 417 AND PAR.C99996 = 0 AND YEAR(PAR.C99992)
= 9999 AND YEAR(PAR.C99995) = 9999 AND YEAR(VER_OBJ.C99995)=9999 AND YEAR(VER_OBJ.C99992)=9999
AND YEAR(MFZ.C99995)=9999 AND YEAR(MFZ.C99992)=9999 AND VER.C95836 = 1 AND VER.C99996 = 0
AND YEAR(VER.C99995) = 9999 AND YEAR(VER.C99992) = 9999 AND VER.D95893 IN (24,25,26,27) ),
partner_all (PNR, Number) as (SELECT PAR.C97251 AS PNR, COUNT(*) FROM DB2PVIEW.VGESGPA2 PAR, DB2PVIEW.VALL0800 VER WHERE PAR.C95826 = VER.C95826 AND PAR.C97150 = 417 AND PAR.C99996 = 0 AND YEAR(PAR.C99992) =
9999 AND YEAR(PAR.C99995) = 9999 AND VER.C95836 = 1 AND VER.C99996 = 0 AND YEAR(VER.C99995) =
9999 AND YEAR(VER.C99992) = 9999 GROUP BY PAR.C97251) select make, avg(number) as number_of_contracts, count(*) as number_of_customersfrom Partner_mfz, partner_allwhere partner_mfz.pnr=partner_all.pnr group by make having count(*) > 1000order by 2 desc
17DB2 IDAA
10100
100010000
Response time in sec
Real-Time Analysis: RisksUnvalidated correlations
18
Both statistics, number of storks observed, and number of new-bornchildren counted in same area and year, are correlated with time, and both were decreasing, independent from each other.
No. of new-born children in same area and year
No. of storksobserved
19
Agenda
Swiss Mobiliar in a Nutshell Analytics on OLTP data – where we are today Analytics on OLTP data – PoC with Database In-Memory PoC Database In-Memory Results Further Experiences and Next Steps with Database In-Memory at Swiss Mobiliar Summary
20
Oracle Database In-Memory PoCObjectives
The PoC (Beta-test) of Oracle 12c Database In-Memory shall prove that
comparable, if not better, response times might be achieved for the same data and the same query compared with DB2/Netezza.
the number of queries with demand for manual query statement tuning will reduce by at least 90%.
analytics on OLTP data will be, after migration DB2Oracle,
• very efficient.
• transparent for users and applications.
• without impact on OLTP processing.
21
Oracle Database In-Memory
21
column-oriented
Memory
data
Memory
row-oriented
data
So far: Data organized in rows, data loaded into memory at first usage
designed for transactions
New: Additional: Data organized in columns, permanently stored in memory
designed foranalytics
automated, near real-time
replication
22
Offer and Contract data Migration DB2Oracle of 97 tables with No. of rows between 0 and 2.4 billion Offer data (550K contracts in progress) Contract data (5.4M contracts)
In other words: Every 10th contract is in progress.
This data is not part of the traditional data warehouse.
Analytics of this data needs to be close to real time
All tests were performed by applying 13 real user reports with real data.
Oracle Database In-Memory PoCConfiguration
23
1. How expensive is the migration of the DB2 data into Oracle?
2. How much tuning effort was required after activating Oracle Database In-Memory?
3. Are the response times better than with DB2? Which part of the improvement is due to Database In-Memory?
4. How much administration effort is necessary to deploy and run Database In-Memory?
5. Which are the prereq‘s to apply Database In-Memory?
6. Which are the strengths and weaknesses of Database In-Memory?
Oracle Database In-Memory PoCThe PoC should answer the following 6 questions
24
Agenda
Swiss Mobiliar in a Nutshell Analytics on OLTP data – where we are today Analytics on OLTP data – PoC with Database In-Memory PoC Database In-Memory Results Further Experiences and Next Steps with Database In-Memory at Swiss Mobiliar Summary
25
The SpeakerPaolo Kreth
Born 1968 in Genoa, Italy Master Degree in Informatics at Genoa University Working with Oracle since 1998 – Oracle 8i Object Relational Database Since 2010 OracleTechnical Lead @Mobi 1968 – Ferrari 365 GTC Coupé
26
Question 1: Database migration from DB2 to Oracle
• Data was successfully migrated from DB2 to Oracle: 3 days effort.
• Adjustments from DB2 to ORACLE- TIMESTAMP Data Type conversion 24:00 23:59:999999
(Oracle does not recognize 24:00)- Adjustments on DDL-definitions for empty Strings/Numbers/Timestamps (DB2)
CREATE TABLE TEO_TBENPRO
( C43087 CHAR(30) DEFAULT ON NULL ' ‘ NOT NULL, C…. NUMBER(30) DEFAULT ON NULL 0 NOT NULL, …. )
• No runtime errors.
27
Question 2: How much tuning effort was required after activating Database In-Memory?
• No tuning was required.• One single optimization:
- We changed the partitioning layout of a table to optimize it for the required report (Amount: 1 hour)
28
Question 3 : Comparison Oracle Native – Oracle In-Memory
• Comparison based on Elapsed Time: … Oracle (native) vs. Oracle Database In-Memory
0.10
1.00
10.00
100.00
Oracle native
Oracle IM Column Store
Total elapsed time (s):
ORACLE native 194.21
ORACLE IM 44.7
29
Question 3: HW-Comparison between IDAA and Oracle
IBM Netezza IDAA vs. Oracle 12c Database In-Memory
… in the red corner….
Oracle 12c mit OIMO:• Hitachi Unified Compute
Platform • 16 Cores• 384 GB RAM
• no attache PCI Flash Disks• OS: RedHat 6.4• Oracle 12.1.0.2 Beta 3
… in the blue corner….IBM Netezza IDAA:• PureData System for Analytics
N1001-002 • 24 Cores• 72 GB RAM
• Netezza V 7.02• IDAA V3
Ready to Rumble !!!!!!
30
0.10
1.00
10.00
100.00
IDAA
Oracle IM Column Store
Question 3: Oracle 12c Database In-Memory vs. IBM Netezza
• Comparison of elapsed time … … Netezza IDAA vs. Oracle Database In-Memory
Total elapsed time (s):
IDAA 76.74
ORACLE IM 44.73
31
Question 4: Adminstration of Oracle Database In-Memory
• 50 GB RAM was reserved for the in-memory column store- …41 GB used (Memcompress for Query)- …on SAN-these data occupied 170 GB. Compression Ratio : about 4.15
• Data modifications were propagated near real time to the Column Store in our test.
• Oracle Database In-Memory allows us to drop all „non-unique“ Indexes
• We did not have to load the whole table into memory, we loaded only selected partitions or even columns.
32
Question 5: Pre conditions for optimal usage
Which are the pre conditions for an optimal usage of Oracle Database In-Memory?
a) Use of ORACLE built in functionalities to restrict the data set (Partitioning)- The optimal partitioning method can significantly improve the query performance
b) Find the optimal degree of parallelism in 12c - In our tests we found that the optimal degree of parallelism was 8
(this value depends on hardware we tested on – the CPUs were unable to get more data from memory)
- More parallel slaves produced only more overhead- For different hardware this value could differ
c) Plan Stability- Of relevant importance is the stability of execution plans. Therefore the optimizer
statistics have to be up to date.
33
Question 6: Strengths and Weaknesses of Database In-Memory
Strengths:• It scales very good incrementing the degree of parallelism. Also small degree increments
show big performance improvements.• Read and Filter of large data volumes• Easy administration
You can improve the performance of an application within minutes
Weaknesses:• Sorts of large data volumes is still a bottleneck ( 100+ GB)
Sorts still use temporary tablespace
• Proposal to ORACLE:
Temp Segment in Memory for Column store ? Feedback to Oracle has been provided.
34
Agenda
Swiss Mobiliar in a Nutshell Analytics on OLTP data – where we are today Analytics on OLTP data – PoC with Database In-Memory PoC Database In-Memory Results Further Experiences and Next Steps with Database In-Memory at Swiss Mobiliar Summary
35
Results within the MobiliarFIS Project
What were the straits within the MobiFis Project: MobiFis eDWH is still in developmentWe could work only on a small set of queries and
reports Solution:
We did the tests only on Queries on a fact table and the
corresponding dimension tables Beside the small query set on top of the reports we got from development we built
synthetic queries trying to simulate future reports by creating: Sums, averages and other group functions on the existing tables Grouping of values
We issued these queries on the following data volumes• 1 Fact-Table with1.9 Mrd. Rows• Up to10 Dimension Tables between 100 and 2.5 Mio. Rows
36
Results on MobiliarFIS
Comparison of elapsed times
Query Elapsed Time(s)
11g 12c 12c In-Memory
Overview Tb Level (BO-Report) 949 183 135
Numer of contracts for one product in 01/14 32 29 4
Building blocks per single contract 274 306 0,3
Sums on single contracts 270 310 0,053
Sums and group by on single contracts 289 245 0,5
All contracts for a product 1 0,045 0,018
Sum 1815 ~ 1073 ~ 140
Factor 1 ~ 1.7 ~ 13
37
Resume MobiliarFIS
Summary
• The Results from the PoC on Oracle 12c "In-Memory Column Store" showed an improvement of one or two magnitudes.
• Typical DWH queries like sums, averages etc. can be improved dramatically .
• The assumptions made at the start of the PoC could be confirmed.
Next Steps
• MobiliarFis has been migrated on 12.1.0.2 and we are beginning to develop reports which will benefit from Oracle Database In-Memory.
38
Test on an existing Application - RICO
We tested on the database of our risk controlling application (RICO)
The Rico Database occupies about 1 Tb of Data
The Rico application is over 20 Years in production
We collected the most important queries from the user‘s perspective
These queries were the candidates for improvement.
39
Tests on RICO – Before starting
We analyzed the existing application and investigated performance optimization we could get before 12c:
Compression
Optimization of PCTFREE parameter
Change of the partitioning schema of some tables
We started using In-Memory after these changes
40
Results RICO
On average the queries were running 2.5 times faster.
Parallel Degree Query 1 Query 2 Query 3 Query 4 Query 5 All
1 3.53 1.82 0.78 2.52 2.32 2.57
2 2.66 0.94 5.82 2.42 1.63 2.23
4 2.87 1.06 5.49 2.39 2.44 2.56
8 2.65 1.40 5.89 1.95 2.70 2.44
16 2.72 1.73 6.41 2.79 2.73 2.75
32 2.69 2.16 7.79 2.74 2.74 2.73
Speed Up Oracle 12c (without In-Memory) vs. Oracle 11g
41
Results RICO
Speed Up Oracle 12c In-Memory vs. Oracle 12c
Parallel Degree Query 1 Query 2 Query 3 Query 4 Query 5 All
1 108.89 25.31 48.77 38.50 46.12 46.89
2 138.89 4.66 6.25 51.20 49.80 49.10
4 261.17 4.81 5.67 88.12 87.58 86.15
8 280.25 4.28 4.67 154.45 144.61 135.51
16 370.25 3.46 3.40 212.10 233.79 191.44
32 404.91 2.58 0.32 262.35 234.53 130.24
On average the queries were running between 46 and 191 times faster.
42
Results RICO
Both slides combined: Speed Up Oracle 12c In-Memory vs. Oracle 11g
Parallel Degree Query 1 Query 2 Query 3 Query 4 Query 5 All
1 384.04 46.16 37.84 96.91 107.03 120.40
2 370.11 4.39 36.38 124.10 81.15 109.43
4 750.78 5.13 31.11 210.42 213.33 220.82
8 743.44 6.00 27.50 301.76 390.77 330.33
16 1007.25 6.00 21.80 591.14 638.32 525.94
32 1087.73 5.58 2.48 717.65 642.95 354.92
On average the queries were running between 109 and 525 times faster.
43
Summary RICO
Rico has been migrated on July on 12.1.0.2
A development environment has been built to test the upgrade and the effects of In-Memory
We are using the SQL Performance Analyzer of the Real Application Testing option to identify improvements and regression
We are involved in the Oracle 12c Reference Program and we are working closely with Oracle.
Oracle In Memory will be activated after the Mobiliar Software Release by end of October
44
Agenda
Swiss Mobiliar in a Nutshell Analytics on OLTP data – where we are today Analytics on OLTP data – PoC with Database In-Memory PoC Database In-Memory Results Further Experiences and Next Steps with Database In-Memory at Swiss Mobiliar Summary
45
Oracle Database In-Memory Benefits End User Perception
Like an additional, fast, access path
Fastest way from San Francisco to Chicago?
SQL Tuning DB In-Memory
Beam me up, Scotty31 hours 4h 30min(+1h 30min)