Db2 Analytics Accelerator · 2020-03-24 · Data and AI on IBM ZData and AI on IBM Z Db2 Analytics Accelerator Version 7.5 Integrated Synchronization Integrated, low-latency data

© 2019 IBM Corporation1

Data and AI on IBM Z Data and AI on IBM Z

Db2 Analytics AcceleratorProvider of Data Gravity

Ed Lynch

IBM z Data & AI Presales Engineer

[email protected]



Agenda

• Db2 Analytics Accelerator Overview

• Features of IDAA

• Approaches to Using IDAA

• Benefits

Db2 Analytics Accelerator



Traditional Approach to Analytic Systems

Operational

Applications

Transaction Processing

Shared Everything DB

High volume business transactions and batch

processing running concurrently

Analytic Applications

Data Store, Business

Intelligence, Predictive

Analytics

Shared Nothing DB

/ Columnar DB

Low volume complex

queries and batch

reporting

Latency?

Security?

Data Governance?

Complexity?

Data transfer



The Hybrid VisionDelivering Business Critical Analytics

Transactional Processing, Traditional Analytics &

Business Critical Analytics

Hybrid DB

High volume business transactions and batch

reporting running concurrently with complex

queries

Combined Workloads

Reduced Latency. Greater Security.

Improved Data Governance. Reduced Complexity.



Take the analysis to the data

• Avoid all the pitfalls of moving the data

Simplified infrastructure with more resiliency

• One copy of the data not dozens

Much more secure

• Z security built in

Lower cost

• Saves money (Infrastructure, SW, and people)

Much lower analytics latency

• Low to no latency with transactional data

A Data Gravity approach performs analytics

where the majority of the data originates

By far, the best place to analyze

Z data is on IBM Z

Db2 Analytics Accelerator and Data Gravity Benefits



Db2 Analytics Accelerator and Db2 for z/OS

WHATAn integrated, hybrid workload-optimized database management system

HOWRuns each query workload efficiently in its optimal environment

WHYTo ensure the greatest performance and cost efficiency

TransactionProcessing

HTAP AnalyticalWorkload

WOWExploit IBM Z data in-place to improve efficiency, drive smarter outcomes and gain competitive differentiation



Deep DB2 integration within z SystemsApplications

Application Interfaces(standard SQL dialects)

DBA Tools, z/OS Console . . .

Operational Interfaces(e.g. DB2 Commands)

Data

Manager

Buffer

ManagerIRLM

IBM DB2

Analytics

Accelerator

. . .Log

Manager

DB2 for z/OS

Superior availability,

reliability, security

Workloadmanagement

z/OS onz Systems

Superior

performance

on analytic

queries



Query execution process flow

Ac

ce

lera

tor D

RD

A R

eq

ue

stor

ApplicationInterface

Heartbeat (availability and performance indicators)

Application

Optimizer

Query execution run-time for queries

that cannot be or should not be

routed to Accelerator

Heartbeat

Queries executed

with Accelerator

Queries executed

without Accelerator



Powered by Db2 with BLU Acceleration (Db2 Warehouse)

• Fast ingest for incremental updates, and thereby low HTAP query delay!• IBM’s premier analytics engine across many products• Latest analytics technology innovations• SQL compatibility across all IBM products• High degree of concurrent users and queries

In-memory column processingwith dynamic movement of data

from storage

Multi-core and SIMD parallelism

(Single instructionMultiple Data)

Patented compression technique preserves order -- data can be used without decompressing

Skips unnecessary processing of irrelevant data



SQL functionality support and restrictions

•Improved Db2 for z/OS SQL support on Accelerator V7 (compared to V5)• All data types supported except LOBs or XML

• Improved Db2 for z/OS function support on the Accelerator

• Still some not supported, e.g ACOS, ASIN, CLOB, ..

• Correlated subquery support

• Recursive SQL support

• Special register support

•Restrictions: • No user defined functions (except inline SQL scalar UDF, compiled SQL scalar UDF)

• No multiple encoding schemes in the same statement

Knowledge Center: Conditions for query routing to an acceleratorhttps://www.ibm.com/support/knowledgecenter/en/SS4LQ8_7.1.0/com.ibm.datatools.aqt.doc/gui/concepts/c_idaa_query_offloading_criteria.html

https://www.ibm.com/support/knowledgecenter/en/SS4LQ8_7.1.0/com.ibm.datatools.aqt.doc/gui/concepts/c_idaa_query_offloading_criteria.html



Pass-through support for Db2 Warehouse built-in functions Enhancing Db2’s native SQL Capabilities with the Accelerator

• Many built-in functions that are supported by the underlaying DBMS (Db2 Warehouse) in the Accelerator are not supported natively by Db2 for z/OS (yet).

• Some of them can now be used in SQL queries routed to the Accelerator with the new Built-In-Function (BIF) Pass-through support

• Db2 for z/OS is ”aware” of the Accelerator, when parsing the SQL statement.

• If a BIF is referenced, which is only available on the Accelerator, the Db2 for z/OS parser validates the signature and allows its invocation within the rewritten SQL.

• Db2 for z/OS still needs to validate parameters, return types, …. Therefore the pass-through is limited to commonly requested BIFs.

• Supported BIFs

• OLAP/Aggregate functions: CUME_DIST, FIRST_VALUE, LAG, LAST_VALUE, LEAD, NTH_VALUE, NTILE, PERCENT_RANK, RATIO_TO_REPORT

• Scalar functions: REGEXP_COUNT, REGEXP_INSTR, REGEXP_LIKE, REGEXP_REPLACE, REGEXP_SUBSTR

• Db2 12 only, FL504



Agenda




• Benefits




Accelerator on IBM Integrated Analytics System

• Pre-configured hardware and software for easy deployment, management, and high performance

• Secure, flexible and elastic data storage –easy to deploy and manage

Accelerator on IBM Z

• Deep integration with IBM Z offers a unified homogeneity of service, support and operations

• Flexible capacity to respond to peak analytic workload requirements

Flexible, integrated deployment options

Db2 Analytics AcceleratorHigh-speed analysis of enterprise data for real-time insight

Uniform experience – transition easily between deployment options with one API and one database engine



Db2 Analytics Accelerator Version 7.5

Db2 Analytics Accelerator Version 7.5 delivers:

• Integrated Synchronization a new advanced data synchronization technique

• A wider range of scalability for Db2 Analytics Accelerator on IBM Z deployments -- from very small to very large

General Availability: December 6, 2019



Incremental update architecture with CDC

Applications executing I/U/D Statements on replicated tables

Accelerator Users enabling tablesfor replication

CDC Capture

Agent

Table T1

CDC

Replication

Engine

Db2 Log Data

Table T2

Table T3

Table T1

Table T2

Table T3

CDCAccess Server

Accelerator

Server

Manages Subscription

T2

Committed Data

T3T1

Stored Procedures



Db2 Analytics Accelerator Version 7.5Integrated Synchronization

Integrated, low-latency data coherence protocol between Db2 for z/OS and the Db2 Warehouse

• zIIP enabled

• Complete application transparency

• Enterprise-grade HTAP enabler

• Simplified administration, packaging, upgrades, support . . .

Deeper integration between Db2 for z/OS and Db2 Analytics Accelerator to provide insight from the most current transactional data



Integrated Synchronization - Db2/Z to-Accelerator data synchronization

Applications executingI/U/D Statements on replicated tables

Accelerator Users enabling tables for replication

Table T1

Log data

processor

Db2 Log

Table T2

Table T3

Table T1

Table T2

Table T3

Accelerator

Server

Encrypted Log Data

Stored Procedures

Log Data

Provider Staging

area

Process control



Why True HTAP?

• Single System, Single Interface

• Data coherency between Db2 and Accelerator

• Don’t worry about latency of committed data

• Queries run first when data available

• No longer need to explain to users why committed data not seen.Data used by SELECT contains corresponding commits

• e.g. Use case ”Fraud Detection”Detection query works on most recent data

• e.g. Use case “Multi form consistency”Enter data in a field on a panel. Application commits data.

=> Just entered data shown on following panels

21



How does HTAP work?

Wait for committed data

from time of SQL request

22

Asynchronous

replication

Most

recent

committed

data

available?

no

Wait for

given

time

period

Most

recent

committed

data

required?

yes

no

Initiate

apply

Write

requests

OLTP

reads

OLAP

readsyes



High Performance Storage SaverStoring historical data in Accelerator only

Support for partitioned-by-range tables, for example:

Time-based partitions, where only the recent partitions are used in a transactional context

(frequent data changes, short running queries), but the entire table is used for analytics,

regulatory/audit purposes, etc. (data intensive, complex queries)

Active

Part #1Part #1

Part #2 Part #3 Part #4

Part #5 Part #6 Part #7 HistoricalNo longer present on

DB2 Storage

Query from Application to either Active only or Active and Historical



Non-accelerator DB2 table

• Data in DB2 only

Accelerator-shadow table

• Data in DB2 and the Accelerator

Accelerator-archived table / partition

• Empty read-only partition in DB2

• Partition data is in Accelerator only

Accelerator-only table (AOT)

• “Proxy table” in DB2

• Data is in Accelerator only

Accelerator-only table type in DB2 for z/OSCreation (DDL) and access remains through DB2 for z/OS in all cases

Table 1

Table 4

Table 3

Table 2

Table 4

Table 3

Table 2



Accelerator-only tablesSupporting in-database transformation and multi-step processing

Introduction of Accelerator-only tables (AOT)

to store intermediate or final results of data

transformation or reporting processes

▪ Can be used to store a set of data in Db2 Analytics Accelerator only, not on Db2 for z/OS, without using the High Performance Storage Saver functionality

▪ Accelerate in-database data transformations and data movement processes

▪ Reduced need of data movement processes to other platforms for data transformation purposes

▪ Enables multi-step reporting on the Accelerator

▪ Saves disk space and CPU cost on IBM Z currently used for transformations and reporting steps

▪ Allow data preparation steps for data mining and other advanced analytics to execute on the Accelerator



Multi-step reporting applications with DB2 for z/OSWith Accelerator-only tables: Temporary objects and processing on the Accelerator

Temporary results

2Temporary results

1Temporary results

Reports and Dashboards

n

Multi-Step Report

n

1

2

Reporting Application

Credit Card

Transaction History

Customer

Summary Mart

Credit Card

Transaction History

Customer

Summary Mart

Data for transactional and analytical processing



Using Accelerator-only tables and ELT logic in the Accelerator(In-database transformation)


Systems (OLTP)

Analytics

Advantages: • Simpler to manage

• Better performance and reduced latency Data for transactional and analytical processing

Customer

Transactions

Customer

Data

Customer Transaction Summary and History AOTs

Customer Summary

Mart AOTs

Customer

Transactions

Customer

Data

ELT logic



Data scientist work areaUsing Accelerator-only tables for ad-hoc analysis


Systems (OLTP)


Customer

Transactions

Customer

Data

Customer

Transactions

Customer

Data

Work database John

Work Area AOTs

Work database Bob

Work Area AOTs

Data Scientist John

Data Scientist Bob



Integrate more data sources for analyticsLoad external data to the accelerator and combine it with data from Db2 for z/OS for reporting and analytics


Systems (OLTP)

Analytics


Customer

Transactions

Customer

Data

Combined result in AOTs

Customer

Transactions

Customer

Data

Related data from

other sourcesExternal

Files

Imagecopy

DB2 Analytics Accelerator Loader

VSAM &

Flat FileOracle

IMSOther

Relational

non z



Db2 Analytics Accelerator Loader for z/OS*

• Significant cost & time reduction by eliminating manual ETL processes of non-Db2 data

• Insight into more data types such as IMS, VSAM, sequential files, Oracle, Adabas, etc.

• Ensure High Availability for critical analytics applications

• Protect critical analytics data with backup & restore strategy

• Support cascading changes made to replicated tables to reduce data latency

• Enhance use of Accelerator-only tables with one-step load and load resume

Extend analytic capabilities by bringingnon-Db2 data to IBM Db2 Analytics

Accelerator and IBM Z

*Separately licensed option



Connectivity options

▪ residing in the same LPAR

▪ residing in different LPARs

▪ residing in different CECs

Full flexibility for DB2 systems:

Multiple DB2 systems can connect

to a single Accelerator

A single DB2 system can connect

to multiple Accelerators

Multiple DB2 systems can connect

to multiple Accelerators

▪ being independent (non-data sharing)

▪ belonging to the same data sharing group

▪ belonging to different data sharing groups



Federation support

• Without Federation: Data from Db2 subsystems sharing same accelerator is fully isolated

• With Federation: Grant read access to a table from another Db2 subsystem

• Prerequisites:

• Data from different DB2 systems is loaded into the same accelerator

DB2R DB2O1….

Accelerator

SELECT

….

DB2O2

….



Agenda




• Benefits




Typical Approaches – The “Blue Birds”

• Accelerate Existing Queries

▪ Accelerate Existing Complex Queries – static & dynamic (e.g. QMF, Cognos,..)

▪ Run Costly Queries More Frequently -> Increased Business Value

▪ Run “Abandoned Queries” -> New Business Opportunity

• Cost Savings

▪ CPU Savings & Reduced Elapsed Times

▪ Db2 Index Elimination &/or Reduction – storage savings

• “Online Archive” via High Performance Storage Saver Feature

▪ Save on DB2 storage, data maintenance, and data access costs

▪ Immediate Access – zero recall time of archived data

▪ Instantaneous & Simultaneous Access to Current & Archived Data



Typical Approaches – “Effort Required”

• In Accelerator Database Analytics – go from ETL to ELT using AOTs

• Single Centralized Hybrid Data Store

▪ Shift Query Workload back to Db2 for z/OS with IDAA

▪ Eliminate costs of separate DWH, ODS, and DataMarts

▪ ETL / Replication Cost Elimination

▪ Have seen 20-40% of zCPU Burned for ETL and Replication Purposes

▪ Convert “Transactional Batch Workloads” and Batch Jobs that perform “Application Joins” to use IDAA eligible queries▪ Requires consolidating multiple SQL calls into single calls

▪ Reduction of supporting Application code



Agenda




• Benefits




Productivity Improvements

Productivity

ImprovementBenefits of Time Savings▪ Dramatic Query Elapsed Time Reductions

− Saved Time for End Users

− Improved Customer Service/Support

− End User Satisfaction

− Revenue Growth Opportunities

▪ Online Archive – No Waiting for Recall, Access to Active & Archive

▪ Administration Time Savings

− Fewer Database Maintenance Jobs to Create/Maintain

− Faster Maintenance Execution

− DBA Time Savings

− Less Time Analyzing & Tuning Queries, & Testing



Cost Reduction Opportunities

▪ Infrastructure Simplification – Eliminate/Reduce

− Same Tooling (Dev/Test/Admin/Monitoring)

− Security of System z

− ETL & ETL Creation/Maintenance Costs

− Replication

− Eliminate DataMarts (system, storage, maintenance)

− Eliminate Network Impact of Data Movement

− Fewer Copies of Data = Reduced Storage Costs

− Lower Latency = Improved Business Value

− Fewer Systems to Provision, Operate, Maintain, Power, Cool, &

Provision

− Simplify Data Governance

Cost

Reduction



New Business Opportunities

New Business

Opportunities

For Customers & Customer Service

▪ Operational Analytics

▪ Real-time Business Dashboards

▪ Real-time Reporting

▪ Reduced Time to Market

− Leveraging Existing Transactional Data

− Don’t have to Wait to Design DWH or Data Lake

− Don’t have to Wait to Build DWH or Data Lake

▪ Low Latency – Improved Business Value

▪ Run Workloads More Frequently

▪ Brush Dust off of “Shelved Projects”- Shift in Cost Equation



Learn more!

• What’s available?• Product videos

• Guided demo

• Hands-on lab

Visit the Db2 Analytics Accelerator on IBM Demos: http://ibm.biz/Acceleratordemos

http://ibm.biz/Acceleratordemos



Summary




• Benefits




Appendix – Deployment Options Detail



Db2 Analytics Accelerator V7.5, deployment on IBM Integrated Analytics System (IIAS)

• Hardware appliance

• A full solution that provides all components out of the box – including optimized hardware and software

• All components provided by IBM in a balanced, performance-optimized configuration

• HW, which includes the rack, the physical servers and the storage

• SW stack including the Linux operating system, the Docker software as well as the Docker container and the infrastructure management

• IBM Power hardware for the appliance, balanced and optimized for price/performance



IBM Db2 Analytics Accelerator on IIAS Product components

OSA-Express

10 GbE

CLIENT

• Data Studio with Db2 Analytics Accelerator Studio Plug-in

• Data Server Manager 2.1.5 or higher

IBM Z

Db2 for z/OS enabled forIBM Db2 Analytics Accelerator

incl. Stored procedures

IBM Db2 Analytics

AcceleratorUsers/

Applications

Dedicated highly availablenetwork connection

IBM Integrated Analytics System

OSA-Express

10 GbE

Pa

tch

Pa

ne

l



IBM Integrated Analytics System Configurations

IBM Power 8 S822L 24 core server 3.02GHz

M4002-0031/3 Rack

M4002-0062/3 Rack

M4002-010Full Rack

M4002-020Double Rack

M4002-040Quad Rack

Servers 3 5 7 13 25

Cores 72 120 168 312 600

Memory 1.5 TB 2.5 TB 3.5 TB 6.5 TB 12.5 TB

Flash Storage Capacity

27 TB 54 TB 81 TB 162 TB 324 TB



Db2 Analytics Accelerator Version 7.5, deployment on IBM Z

• A software appliance running on IBM Z

• Packages the SW stack into an IBM Secure Service Container to deliver a fully self-managed appliance running in a SSC LPAR that can be deployed in minutes

• Integrates seamlessly into the customer’s Z environment and leverages known LPAR-, memory and CPU management procedures, including call home support for enterprise hardware components.

• Uses customer-provided storage to hold the accelerator-side data

Existing

Co

mp

on

en

ts

SE / HCM

PR/SM LPAR CPU Memory

Storage (SAN)Filesystem

IBM Secure Service Container

Customer’s Storage

Management

Docker container

Db2 Warehouse engine

Authentication

Acceleratorserver

WorkloadMonitoring

Systems Manager

Additional future functionality

Docker supported OS + management

De

livere

d as p

art of

Acce

lerato

r Solu

tion



Db2 Analytics Accelerator on IBM Z

A fully integrated software appliance• Running on IBM Z servers in a Secure Service Container (SSC) LPAR• ”download & go” – very easy to deploy• Delivers the full SW stack including operating system,

docker engine, Db2 Warehouse engine, IDAA components, and infrastructure management

A solution leveraging IBM Z infrastructure• Runs on all IBM Z systems starting with z13 (z13, z13s, z14, z14 ZR1,

z15)• Leverage “white space” (unused capacity) in existing systems – or

dedicate a stand-alone system, e.g., LinuxONE• Leverage existing storage systems (CKD or FB) – no dedicated storage

system required• Leverage existing Z high-availability and disaster recovery solution by

integration of the accelerator with GDPS/Metro



Db2 Analytics Accelerator on IBM Z

• Leverages IBM Secure Service Container− SSC security features ensure that the appliance image cannot be tampered with

and the appliance code and data are protected and kept confidential both in flight and at rest

• Accelerator on Z runs natively in an SSC LPAR on IFLs• Customizable configuration and highly flexible scaling− Single-Node: Minimum 2 IFLs / 64 GB memory, maximum 40 IFLs / 4,096 GB

− Multi-Node: Minimum 30 IFLs / 1.5 TB memory, maximum 190 IFLs / 20 TB

− Can utilize shared infrastructure such as network or storage adapters• No additional licensed software required – no z/VM, no KVM,

no Linux on Z, no Docker, no …− Accelerator not supported to run under z/VM or KVM control

• No operating system access or maintenance− No system administrator access to appliance possible

− All required updates, e.g., security fixes, component updates, etc., are delivered and installed as accelerator image updates

− All required configuration via administrative UI or configuration files



IBM Db2 Analytics Accelerator on IBM ZProduct components

IBM Z

Db2 codeincluding Stored Procedures

Accelerator Appliance• Can be deployed on the

same CEC as Db2 or on a different one

Appliance UI

• Data Studio with Db2 Analytics Accelerator Studio Plug-in

• Data Server Manager 2.1.5 or higher

Dedicated highly availablenetwork connection

OSA

OSA

OSA

OSA



Accelerator on IBM Z – Deployment Options

50



Multi-Node Deployment – Architecture

51

Storage

Db2 z/OS

Accelerator on Z

network

Head

IDAA serverDb2 WH• Catalog• No data

partitions

HiperSocket

Data 1

Db2 WH• Data

partitions

Data 5

Db2 WH• Data

partitions…

…

LPAR Group with absolute capping

SSC LPAR SSC LPAR SSC LPAR

OSA



Multi-Node Deployment – IFLs & Memory

52

Storage

Db2 z/OS

Accelerator on Z

network

Head

IDAA serverDb2 WH• Catalog• No data

partitions

HiperSocket

Data 1

Db2 WH• Data

partitions

Data 5

Db2 WH• Data

partitions…

…

LPAR Group with absolute capping

SSC LPAR SSC LPAR SSC LPAR

OSA 30 IFLs(shared)

256 GBweight=high

14 IFLs(shared)

512 GBweight=low

14 IFLs(shared)

512 GBweight=low

Performance goal:70-80 IFLs comparable to N3001-010

70 IFLs



Multi-Node Deployment – Advantages

53

• Scalability of the Accelerator on Z for the most demanding workloads

• Multi-node accelerator can grow from the entry level (30 IFLs) to the largest size using all available IFLs on a system (190 IFLs on IBM z15) without ever reloading the data

• Extremely flexible adjustment of resources (IFLs, memory, storage) to optimize for the actual workload requirements− Even dynamic adjustments (add/remove IFLs, add/remove memory, add storage) are

supported and require only short or even no downtime

− True “capacity-on-demand” without any disruption (for IFL capacity)

• Maintains all advantages of the deep integration into the Z platform



Multi-Node Deployment – Q&A

• Why are multiple LPARs used instead of just increasing LPAR size?➢ Current limitation of a single SSC LPAR to 128 IFLs and 10 TB memory

➢ Performance measurements have shown that the performance/throughput for a multi-LPAR configuration is better per IFL than scaling a single LPAR far beyond 40 IFLs

➢ Huge LPARs could create issues in operations / maintenance

• Does the Z system running the accelerator need to be dedicated?➢ No. Other workloads can run on the same system. However, all of the accelerator LPARs need to run on the same

system – which may mean a significant portion of the overall capacity.

➢ A single system can also run multiple accelerators – potentially using a mix of both deployment options (single-/multi-node) in the same environment

• Do all LPARs for the multi-node accelerator need to run on a single CPC?➢ Yes. Distribution of LPARs to multiple CPCs is not supported.

• When should I consider the multi-node setup over single-node?➢ If the accelerator sizing indicates resource requirements beyond 30 IFLs and/or 2 TB memory

➢ If growth projections indicate that the single-node capabilities would be exceeded “soon”

• Is there a migration path between single-node and multi-node deployment?➢ No. Such a change requires a new accelerator installation.

54



Documents

Db2 Analytics Accelerator · 2020-03-24 · Data and AI on IBM ZData and AI on IBM Z Db2 Analytics Accelerator Version 7.5 Integrated Synchronization Integrated, low-latency data