IBM DB2 Analytics Accelerator: Real-Life Use Cases Analytics Accelerator - Real Life... · IBM DB2 Analytics Accelerator: Real-Life Use Cases ... analysis to all users in the bank

© 2016 IBM Corporation

IBM DB2 Analytics Accelerator:Real-Life Use Cases

Patric BeckerIBM BoeblingenLaboratory

© 2016 IBM Corporation2

Legal Disclaimer• © IBM Corporation 2016. All Rights Reserved.

• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.


Agenda

§ Value Proposition and Use Cases

• Query Acceleration

• High Performance Storage Saver

• In-database transformations

• In-database Analytics

• High Availability configurations

• DB2 Analytics Accelerator Loader V2.1

• Summary


Hybrid transaction/Analytical processing

The hybrid computing platform on z Systems

Supports transaction processing and analytics workloads

concurrently, efficiently and cost-effectively

Delivers industry leading performance for mixed workloads

The unique heterogeneous scale-out platform in the industry

Superior availability, reliability and security

TransactionProcessing

AnalyticsWorkload


Deep DB2 integration within z SystemsApplications

Application Interfaces(standard SQL dialects)

DBA Tools, z/OS Console . . . Operational Interfaces(e.g. DB2 Commands)

DataManager

BufferManager IRLM

IBM DB2 Analytics

Accelerator. . . Log

Manager

DB2 for z/OS

Superior availability,

reliability, securityWorkload

management

z/OS onz Systems

Superior performance on analytic

queries

PureData System for Analytics


DB2 Analytics Accelerator – Usage scenariosHow organizations leverage the Accelerator today

Accelerateexisting

workloadReduceIT sprawl

Derive newbusiness insight

Include external & historical data

Reduce IT sprawl for analyticsIf the data is offloaded to a distributed data warehouse or data mart• Simplify and consolidate complex

infrastructures, low latency, reliability, security and TCO

Rapid acceleration of Existing Business Critical QueriesIf the data is analyzed on the mainframe• Performance improvements

and cost reduction while retaining z Systems security and reliability

Derive business insight from z/OS transaction systemsIf the data is not being analyzed yet

• One integrated, hybrid platform, optimized to run mixed workload

• Simplicity and time to value

Improve access to historical data and lower storage costsIf the analysis is based on a lot of historical data

• Performance improvements and cost reduction


Agenda








• Summary


Halkbank Case Study

“ We wanted to improve the performance for complex analytic queries – both in order to support rapid and accurate decision making, and in order to meet deadlines for regulatory reporting, where we face penalties of tens of thousands of dollars for late reporting. We expect to be able to move 70 percent of our queries to DB2 Analytics Accelerator, accelerating queries, reducing our DB2 licensing costs on the mainframe, and freeing up processing resources for other workloads.

-- Cenk Niksarlı, Head of IT Infrastructure at Halkbank

http://www-03.ibm.com/software/businesscasestudies/us/en/corp?synkey=E944346V10306Q58

Halkbank extends banking services on social and mobile channels securely and cost-effectively


Swiss Mobiliar Case Study and Video

“ DB2 Analytics Accelerator makes it possible for us to execute 90% of our queries 25x faster, and 50% of them 100x faster …

Queries that used to take five hours to complete are now processed in just 20 seconds in the optimized mainframe environment—and we can run them any time, day or night, with no interruption to our production systems on the mainframe.

-- Thomas Baumann, IT Performance Architect at Swiss Mobiliar

Swiss Mobiliar accelerates 50% of queries by a factor of 100

Case Study: http://www-03.ibm.com/software/businesscasestudies/us/en/swzseries?synkey=T281533L28114L28Video: http://www-03.ibm.com/software/businesscasestudies/en/us/swzseries?docid=ASHF-9GFDH9


Banca Carige Case Study and Video

“ DB2 Analytics Accelerator is key in our vision to provide the right level of data analysis to all users in the bank and possibly outside the bank.

By using DB2 Analytics Accelerator, we expect to reach ... all criteria we had in mind and create greater benefits for our users because our data are not moved...

-- Daniel Cericola, IT & Governance at Banca Carige

Case Study: http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=AB&infotype=PM&appname=SWGE_ZS_ZS_USEN&htmlfid=ZSC03256USEN&attachment=ZSC03256USEN.PDF#loaded

Video: https://www.youtube.com/watch?v=N7arHCI-CKk&list=PLIbvgyHNmNr1imFVslskIpUuhBHkmNHr2&index=1

Providing superior customer service on a secure platform


Swiss Re Case Study and Video

“Our users are getting their reports as much as 70 percent faster—reports that took 10 hours are now available the same day, so user satisfaction has increased dramatically,” says Estermann. “Because those reports contain key analytics that guide pricing and decision making across our lines of business, the solution has the potential to sharpen the company’s competitive edge moving.

-- Reto Estermann, Director, Information Technology at Swiss Re

Case Study: http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?subtype=AB&infotype=PM&appname=SWGE_ZS_ZS_USEN&htmlfid=ZSC03128USEN&attachment=ZSC03128USEN.PDF

Video: https://www.youtube.com/watch?v=xkcp_pJxT5E&feature=youtu.be}

Speeding high-level decision making that boosts the bottom line


Agenda








• Summary


High Performance Storage SaverStoring historical data in Accelerator only

Support for partitioned-by-range tables, for example:Time-based partitions, where only the recent partitions are used in a transactional context (frequent data changes, short running queries), but the entire table is used for analytics,

regulatory/audit purposes, etc. (data intensive, complex queries)

ActivePart #1Part #1

Part #2 Part #3 Part #4

Part #5 Part #6 Part #7 HistoricalNo longer present on

DB2 Storage

Query from Application to either Active only or Active and Historical


Agenda








• Summary


In-database transformation and multi-step processing§ DB2 Analytics Accelerator introduces Accelerator-Only tables to store

intermediate or final results of data transformation or multi-step processes§ Accelerator-Only tables enable complex data transformations taking place on DB2

Analytics Accelerator, called “In-database transformation”

§ Accelerator-Only tables enable intermediate and final results of reporting or other processes to be generated and stored on DB2 Analytics Accelerator, called “In-database multi-step processing”

§ Advantages:§ Accelerate in-database data transformations and data movement processes§ Reduced need of data movement processes to other platforms for data transformation

purposes§ Enables multi-step reporting on the Accelerator§ Saves disk space and CPU cost on z Systems currently used for transformations and

reporting steps§ Allow data preparation steps for data mining and other advanced analytics to execute

on the Accelerator§ Store a set of data in DB2 Analytics Accelerator only, not on DB2 for z/OS, without

using the High Performance Storage Saver functionality


Introducing Accelerator-only table type in DB2 for z/OSCreation (DDL) and access remains through DB2 for z/OS in all cases

Non-accelerator DB2 table• Data in DB2 only

Accelerator-shadow table• Data in DB2 and the Accelerator

Accelerator-archived table / partition• Empty read-only partition in DB2• Partition data is in Accelerator only

Accelerator-only table (AOT)• “Proxy table” in DB2• Data is in Accelerator only

Table 1

Table 4

Table 3

Table 2Table 2

Table 4

Table 3


Accelerator-only tables – Technical basics

§ AOTs are created and dropped using DB2 DDL statements (CREATE; DROP)• Accelerator must be started• QUERY ACCELERATION behavior may have any value during CREATE/DROP• Syntax:

§ CREATE TABLE MYTABLE (...) CCSID ccsid IN ACCELERATOR <ACCEL1>;§ DROP TABLE MYTABLE;

§ Recommended to create a database in DB2 to be used for the AOTs • CREATE TABLE MYTABLE (...) CCSID ccsid IN ACCELERATOR <ACCEL1> IN DATABASE MYDB;

• Usual authorization necessary to create objects in database

§ SELECT and INSERT/UPDATE/DELETE operations using AOTs can only run on the Accelerator• QUERY ACCELERATION behavior must be set to ENABLE/ELIGIBLE/ALL• Accelerator-shadow tables, Accelerator-archived tables and other AOTs can be used in

the same statement

§ Dynamic and static SQL can be used with AOTs


Accelerator-only tables – Technical basics

§ INSERT FROM SELECT into an AOT§ Can only run on the accelerator§ All tables in the SELECT must reside in the accelerator

§ Accelerated shadow DB2 tables§ Archive tables§ AOTs

§ QUERY ACCELERATION behavior must be set to ENABLE/ELIGIBLE/ALL

§ Cross-Loader to load data from AOTs into DB2 for z/OS§ Different table names


Performance INSERT from SELECT

19

§ Up to 95% better elapsed time and negligible CPU time in DB2 for INSERT FROM SELECT into accelerator-only tables for large amount of data

0

100

200

300

400

500

600

1 10 100

Class2

CPU

Time(LowerisBetter)

#RowsinMIll

Class2CPUTime

CL2CPU_DGTT CL2CPU_PBG CL2CPU_IDT


Multi-step reporting applications with DB2 for z/OS BEFORE Accelerator-only tables: Report processing on DB2, source data might reside on the Accelerator already

Temporary results

2Temporary results

1Temporary results

Reports and Dashboards

n

Multi-Step Report

n

1

2

Reporting Application

Credit CardTransaction History

CustomerSummary Mart




Multi-step reporting applications with DB2 for z/OSWith Accelerator-only tables: Temporary objects and processing on the Accelerator

Temporary results

2Temporary results

1Temporary results

Reports and Dashboards

n

Multi-Step Report

n

1

2

Reporting Application





Data for transactional and analytical processing


In-database transformationUsing Accelerator-only tables and ELT logic in the Accelerator

Transaction Processing Systems (OLTP)

Analytics

Advantages: • Simpler to manage• Better performance and

reduced latency Data for transactional and analytical processing

Customer Transactions

CustomerData

Customer Transaction Summary and History AOTs

Customer Summary Mart AOTs


CustomerData

ELT logic


Data scientist work areaUsing Accelerator-only tables for ad-hoc analysis




CustomerData


CustomerData

Work database John

Work Area AOTs

Work database BobWork Area

AOTs

Data Scientist John

Data Scientist Bob


• Deeper insight into operational status through faster reportingSupport multistep reporting applications. Third party BI reporting suites, QMF, home grown applications, etc.

• Accelerated campaign tuning for IBM Campaign (Unica)Improved performance for iterative campaign tuning.

• Simplifying data-transformation processesDelivering in-database transformation within DB2 Analytics Accelerator

• Data mart consolidation through flexible data infrastructure Host data marts on z Systems, where the data originates

• Deeper insight into customers and marketsData scientist work area

• Simpler data integration with DB2 Analytics Accelerator Loader for z/OS (load non-DB2 for z/OS data)Assimilate more data sources for analytics to shorten development cycles and speed integration efforts

• In-database analytics to accelerate predictive analyticsImproves the quality of models, speeds calculations and delivers real-time, actionable business processes

What can I do with AOTs?


(Relatively) New IBM Redbooks publication: SG24-8314-00

§ Analytics on z Systems environment • Warehouse concepts• Logical data warehouse• Transformation patterns

§ Accelerator-only tables • Concepts and architecture

§ Use cases enabled by accelerator-only tables and in-database-analytics

§ Multi-step reporting§ Using QMF to store query results and

importing tables§ Accelerating IBM Campaign processing § In-database transformations § Accelerator and accelerator-only table

usage within DataStage§ Accelerator-only tables supporting data

scientists ad-hoc analysis § Integration of more data sources and

archiving for analytics § In-database analytics

http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg248314.html?Open


Agenda








• Summary


In-database analyticsEnable acceleration of predictive analytics applications

In-database analytics enables SPSS/NetezzaAnalytics (INZA) data mining and in-database modeling to be processed within the Accelerator

§ Accelerates SPSS/Netezza Analytics (INZA) data mining and in-database modeling

§ Allows frequent model refreshes to enable adequate scoring

§ Reduced need of data movement processes to other platforms for predictive analytics purposes

§ Supports the full lifecycle of a real-time analytics solution on a single, integrated system, combining transactional data, historical data, and predictive analytics


In-database analytics – Technical basics

§ Only SPSS Modeler 17 or higher is supported, which needs 18 INZA stored procedures

§ dectree - BuildsaDecisionTreemodelbygrowingandpruningatree§ grow_dectree - BuildsaDecisionTreemodel§ predict_dectree - AppliesaDecisionTreemodeltogenerateclassificationpredictions§ prune_dectree - PrunesapreviouslybuiltDecisionTreemodel§ regtree - BuildsaRegressionTreemodelbygrowingandpruningatree§ grow_regtree - BuildsaRegressionTreemodel§ prune_regtree - PrunesapreviouslybuiltRegressionTreemodel§ predict_regtree - AppliesaRegressionTreemodeltogenerateregressionpredictionsfor

adataset§ naivebayes - BuildsaNaiveBayesmodel§ predict_naivebayes - AppliesaNaiveBayesmodeltogenerateclassificationpredictions

foradataset§ kmeans - BuildsaClusteringmodelthatclusterstheinputdataintokcenters.The

centersarecalculatedasthemeanvalueofthenearestinputdatarecords§ predict_kmeans - AppliesaK-meansClusteringmodeltoclusterrecordsofadataset



§ two_step - BuildsaTwoStep Clusteringmodelthatfirstdistributestheinputdataintoahierarchicaltreestructureaccordingtothedistancebetweenthedatarecords,thenreduces thetreeintokclusters.Asecondpassoverthedataassociatestheinputdatarecordstothenextcluster

§ predict_twostep - AppliesaTwoStep Clusteringmodeltoscorerecordsofadataset§ split_data - Randomlysplitstheinputdataintotwoseparatedsubsets§ pmml_model - StoresthegivenanalyticsmodelasPMMLdocumenttoatable§ list_all_models – Lists allmodelswithgivencharacteristics§ model_exists - Checksifthegivenmodelexists.Themodelcanbesearchedinthecurrent

orinanothergivendatabase.§ drop_model - Dropsthegivenmodel.Allmanagedtablesofthismodelarealsodropped§ drop_all_models – Dropsallmodels

For more information about these stored procedures, see the IBM Netezza In-Database Analytics Reference Guide and the IBM Netezza In-Database Analytics Developer's Guide. These PDF files are delivered in a compressed archive called nz-analyticsdoc-v3.2.0, which is available for download from IBM Fix Central.


Generic Wrapper for Remote Stored ProceduresExternal Providers can supply Stored Procedures to execute on an Accelerator

§ DB2 only acts as gateway



§ Example of how an application calls the DB2 for z/OS wrapper stored procedure:

CALL INZA.KMEANS(’MYACCEL', 'model=adult_mdl, intable=TPCH30M.CUSTOMER, outtable=IWATEST.adult_out,id=C_CUSTKEY, target=C_NATIONKEY, transform=S, distance=euclidean, k=3, maxiter=5', ?, '');

Blue = procedure/algorithm to executeRed = Accelerator to run the procedure onGreen = Algorithm parametersOrange = Table information

More information on wrapper stored procedures in DB2 Analytics Accelerator Knowledge Center: http://www-01.ibm.com/support/knowledgecenter/SS4LQ8_5.1.0/com.ibm.datatools.aqt.doc/sp_msg/SPs/sp_idaa_wrapper.html


In-database analyticsData preparation (using AOTs) and SPSS modeling in the Accelerator


With embedded scoring

Advantages: • Allows fast model refreshes• Ensures adequate scoring • Better performance and

reduced latency• Scoring outside accelerator with SPSS Modeler Server

Scoring Adapter for DB2 for z/OS



CustomerData

Customer TxnData Prep AOTs


CustomerData

Modeling

ModelModel

SPSS Modeler


Installation and setup

https://www-01.ibm.com/support/knowledgecenter/SS4LQ8_5.1.0/com.ibm.datatools.aqt.doc/installmanual/concept/c_idaa_inst_analytics.htmlInstallation package can be downloaded from Fix Central

§ Described in Knowledge Center


In-database analytics (Customer Example)Data preparation (using AOTs) and SPSS modeling in the Accelerator

source(select)

process process typenode

modeling nugget score output

Step 1 IDAA Query acceleration

Step 2 Step 3 In-database transformation and analytics

Step 3: Performance Results• Acceleration of Modelling highly depends on data size.

Step 2: Performance Results:• Data Preparation in minutes which was not possible before• Significant acceleration of data preparation : Up to 240x• End-to-end effect : Up to 23x

Step 1: Performance Results• Significant acceleration of select statement: Up to 10x

Data Preparation Modelling Scoring

source datasource process process(select) (select) result data

Your mileage can and will vary !!!


Agenda








• Summary


Workload Balancing with multiple accelerators having the same data

36

DB2Subsystem or DSG Member

T1 T2 T3

T1

T2

T3

T1

T2

T3

Capacity w

eightCap

acity

wei

ght

Automaticworkloadbalancingbased on

capacity weight

Accelerator 1 Accelerator 2


Query routing after failure of an active accelerator

37


T1 T2 T3

T1

T2

T3

T1

T2

T3

Capacity w

eightCap

acity

wei

ght



Mixed setup with a shared accelerator

38

DB2Production Subsystem or DSG Member

T1 T2

T1

T2

T3

T1

T2

Load / Update

T1, T2

DB2Test system

T3

50% Production50 % Test


Mixed setup with a shared accelerator after a failure

39

DB2Production Subsystem or DSG Member

T1 T2

T1

T2

T3

T1

T2

Load / Update

T1, T2

DB2Test system

T3

100% Production0 % Test


Incremental Update with active/standby capture agents

40

Data Sharing GroupDB2 DB2

Dynamic VIPA

LogCache

Last applied LRSN/RBA


CF



Incremental Update after active capture agent has failed

41


CFDynamic

VIPA

LogCache





High Performance Storage Saver with multiple accelerators after failure



archived

ImageCopy

Table

TableTablePartition 1Partition 2Partition 3Partition 4Partition 5Partition 6

Partition 1Partition 2Partition 3Partition 4Partition 5Partition 6



High Performance Storage Saver with multiple accelerators



archived

ImageCopy

Table

TableTablePartition 1Partition 2Partition 3Partition 4Partition 5Partition 6




Example: GDPS PPRC active/active with multiple site workload


GDPS-PPRC

Maximum 20 km

DB2DB2

CF

Data Sharing Group

P P P S S S

Site 1 Site 2


Example: Workload flow for GDPS PPRC active/active

DB2 DB2Data Sharing Group

Queries

Queries


Site 1 Site 2


Example: GDPS PPRC active/active after failure of one site

P P P S S SGDPS-PPRC

Maximum 20 km

Data Sharing GroupDB2DB2

CF


Site 1 Site 2



Site 1 Site 2

Queries

Queries


Example: Workload flow for GDPS PPRC active/active after failure of one site


Agenda








• Summary


Loading non-DB2 Data Sources into AcceleratorProduct Comparison§ IBM DB2 Analytics Accelerator Loader – The user must:

• Extract data from source (IMS, VSAM, Oracle, SMF, etc.)• Convert extracted data to DB2 external load file format• DataStage or other tooling can be used

• Create a DB2 table that matches format of extracted data• Add newly created table to the Accelerator• Construct a DB2 Load utility field specification that describes the input data• Run Accelerator Loader batch job to load data to accelerator

§ Accelerator Loader V2.1– Automates entire process: • User builds a select statement from data source(s) (IMS, VSAM, SMF, Oracle, …)• Automatically creates the DB2 table • Automatically adds table to Accelerator• Automatically extracts specified source data• Automatically converts data to necessary DB2 format (in memory)• Automatically loads data to Accelerator• Automatically enables table for acceleration

Single batch job!


IBM DB2 Analytics Accelerator Loader for z/OS v2.1

z/OS based Server – Direct Load from Data Sources


0

20

40

60

80

100

120

140

160

180

200

Extract * + DB2 Load + Accelerator Load

Extract * + Loader External Load

LoaderDirect Load from Source

189

128

11

60

2161

15

84

Min

utes

The Old vs The New200GB VSAM KSDS

z13 LPAR: 2 CPs 6 zIIPs w/SMTACCEL_LOAD_TASKS = 16

Elapsed Time GCP Time zIIP time

Traditional Approach Loader V1.1 function Loader V2.1

Loader V2.1 for Direct Load from VSAM into Accelerator

* Extract - A bare bones COBOL program was written/utilized for the base Extract


Agenda








• Summary


DB2 Analytics Accelerator Version 5.1Adding new dimensions in functionality to expand use cases

Rapid acceleration of existing Business Critical QueriesAdding application support fortemporary objects (QMF, Multi-step Reporting, IBM Campaign, etc.)

Individual ad-hoc analysis that provides a Data Scientist Work Area

• Insight into now to maximize business opportunities in today’s dynamic environment

Accelerateexisting

workload ReduceIT sprawl

Derive newbusiness insight

Include external & historical data

Reduce IT sprawl for analytics

• Business agility through simplified architecture

Improve access to historical data and lower storage costs

• Simplified access to information – when you need it

Integrate more data sources for analytics, using DB2 Analytics

Accelerator Loader for z/OS to assimilate with IMS data or data

from other sources

In-database transformation to support Data Stage Balanced

Optimization and the consolidation of ETL/ELT

processing in DB2 for z/OS

Derive business insight from z/OS transaction systemsIn-database analytics to accelerate predictive analytics applications; SPSS/INZA data mining and in-database modeling can be processed within the Accelerator• Real-time, actionable business processes

• Environment to efficiently, continuously test and improve analytic results to drive better customer understanding

Deliver Right-Time Analytics to drive better business outcomes


Documents

IBM DB2 Analytics Accelerator: Real-Life Use Cases Analytics Accelerator - Real Life... · IBM DB2 Analytics Accelerator: Real-Life Use Cases ... analysis to all users in the bank