PureData for Transactions Overview - public.dhe.ibm.compublic.dhe.ibm.com/software/dw/puresystems/tech... · © 2013 IBM Corporation PureData for Transactions Overview Fraser McArthur

© 2013 IBM Corporation

PureData for Transactions Overview

Fraser McArthur ([email protected]) Consulting IT Specialist, PureData Ecosystem Team

mailto:[email protected]

2

PureData


Agenda

Information Technology Challenges PureSystems and PureData Introduction PureData for Transactions Technical Details

3

PureData


Technology is the leading force for impacting businessIs IT ready for the challenge?

* 2012 IBM Global Data Center Study

68% of IT operating costs in 2013 will be for management and administration

Only 1 in 5 organizations allocate more than

50% of IT budget to new projects

2013

68%29%

1996

* IDC; Converged Systems: End-User Survey Results presentation; September 2012; Doc #236966

New server spendingPower & cooling costsServer mgmt & admin costs

4

PureData

© 2013 IBM Corporation© 2012 IBM Corporation4

Clients struggle to overcome barriers of time, cost and risk

Top Causes of Project DelaysHardware

Troubleshooting and tuning production environment

Integration, configuration and testing of the infrastructure

Installation, cabling and network access for the environment

SoftwareIntegration, configuration and testing of applications

Integration, configuration and testing of middleware

Configuration, build and deployment of applications

45%

45%

29%

41%

35%

34%

Phase Time (days) Budget

Specify/design 73 - 96 14% - 16%

Procure 57 - 112 19% - 21%

Implement 74 – 93 12%

Configure/test 74 – 80 10% - 11%

Cluster & HA 66 – 104 11% - 12%

Backup 44 – 108 10%

Tune 89 – 98 9% - 10%

Management 67 – 110 9 – 10%

34% of new IT projects (US) deploy late

Typical IT Project Time and Budget

From a commissioned study conducted by Forrester Consulting on behalf of IBM

5

PureData


Let’s view IT from a new perspectiveSystems with integrated expertise and built for cloud

Integrated by DesignDeeply integrate and tune hardware and software

Built-in ExpertiseCapture and automate what experts do

Simplify the Experience

Make every part of the IT lifecycle easier

Get up and running faster: from 4 – 7 months to hours!

Streamline development and operations: months to

minutes (with repeatability)

Optimize ongoing effort: single management console;

no downtime upgrades!

6

PureData


• Workload optimized performance

• Data load ready in hours

• Integrated management

• Automated maintenance

• Single point of support

Delivering Big Data Platform Services

7

PureData


Real Time Fraud DetectionCustomer AnalysisE-commerce

Transaction Processing

Reporting and Analytics

Operational Analytics

Different applications rely on different data processing

7

8

PureData


Transaction Processing

Reporting and Analytics

Scalable Transactional Database

Different data workloads require different solutions

8

Operational Analytics

9

PureData


• Select model– for Transactions– for Analytics– for Operational Analytics– for Hadoop

• Select size• Load data

Data Platform

Delivering Data Services

Simplicity & Optimizationout-of-the-box

10

PureData


IBM PureData System for TransactionsOptimized exclusively for transactional data workloads

Delivering data services for transactions

System for Transactions

Speed Industry leading DB2 performance Database node recovery in seconds1

Simplicity Database deployment in minutes, not hours1 Runs multiple database software versions Handles more than 100 databases on 1 system2

No planned downtime for firmware / OS upgrades1

Scalability Scaling up to 30x3

Designed to expand from small to medium to large configurations with no planned system downtime

Smart Supports Oracle Database apps with minimal

change; supports DB2 applications unchanged Clients have experienced cases of 10x storage

space savings via Adaptive Compression4

Footnotes:1. Based on IBM internal tests and system design for normal operation under expected typical workload. Individual results may vary.2. Based on one large configuration.3. Based on the designed minimum and maximum processor and memory resources required for a single database.4. Based on client testing in the DB2 10 Early Access Program.

11

PureData


Three Configurations Available

Configurations T1500-96Small

¼ Rack

T1500-192Medium½ Rack

T1500-384Large

Full Rack

Chassis 1 1 2

Compute Nodes(16 cores per node)

6 12 24

CPU Cores 96 192 384

# of clusters/ databases 3/ 30 6/ 60 12/ 120

Memory 1.5 TB 3.1 TB 6.1 TB

V7000 Storage Unit(each unit has: 18 x 900GB

HDD, 6 x 400 GB SSD)

1 2 4

V7000 Storage Expansion(each unit has: 18 x 900GB

HDD, 6 x 400 GB SSD)

1 2 4

User Capacity Raw SSD Storage Raw HDD Storage

18.6 TB4.8 TB

32.0 TB

37.2 TB9.6 TB

64.0 TB

74.4 TB19.2 TB128.0 TB

Upgrade Upgrade

12

PureData


Components – PureSystem Manager (PSM)

PureSystem™ Manager (PSM) Provides Integrated management for all

system resources Built-in Pattern Deployment 2 PSMs per rack for redundancy via active /

passive mode and virtual IP-Address Hosts the Optim® Performance Manager VM

42U41U40U39U38U37U36U35U34U33U32U31U30U29U28U27U26U25U24U23U22U21U20U19U18U17U16U15U14U13U12U11U10U09U08U07U06U05U04U03U02U01U

7189 PDU

7189 PDU

7189 PDU

7189 PDU

PSM 2PSM 1

System Console

BNT Top of Rack Switch

Compute ITE

Compute ITE Compute ITE

Compute ITE


V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Compute ITE


System Console System Console to operate the PureSystem

Manager component

Large Rack

13

PureData


Components – Flex System & Compute Nodes

Flex System™ Chassis 1 for Small/Medium and 2 for Large configuration

Each holds up to 12 Compute Nodes (see below)

Back plane with fully redundant Network (En) and Storage connectivity

Flex System™ Compute Nodes Up to 24 Compute Nodes per rack, hot

swappable

Serve as nodes for DB2 pureScale running as CFs or DB2 Nodes with:

Red Hat Enterprise Linux® 6.2 IBM DB2 ESE 10.1 FP1

Storage optimization and WLM DB2 pureScale TSM for backup and recovery


7189 PDU

7189 PDU

7189 PDU

7189 PDU

PSM 2PSM 1

System Console


Compute ITE


Compute ITE


V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Large Rack

14

PureData


Components - Storage


7189 PDU

7189 PDU

7189 PDU

7189 PDU

PSM 2PSM 1

System Console


Compute ITE


Compute ITE


V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Storage System Storewiz v7000 controller and

expansion units

Up to 192 disks in large configuration.

Storage Network Use of multiple switches & physical

connections

Balanced HDD & SSD Setup 48-disk modules (12 SSD + 36 HDD) to

optimize for performance & cost

Speed & Redundancy Setup via RAID 10

4.8 TB SSD and 32.4 TB HDD capacity in a pair

Large Rack

15

PureData


Components – Network Connectivity


7189 PDU

7189 PDU

7189 PDU

7189 PDU

PSM 2PSM 1

System Console


Compute ITE


Compute ITE


V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller

V7000 Expansion

V7000 Controller


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Compute ITE


Network Connectivity Dual 10Gb Ethernet Switches for external

communication.

− 48 ports SFP+, 4 ports QSFP+

Increased Redundancy and Bandwidth through the use multiple switches and physical connections

− Failure of cable or switch leads to reduced bandwidth - not reduced service

Communication Networks are separated for different duties:

− Customer Data Network used by customer applications to connect to database

− Customer Management Network for remote management access

− DB2 pureScale Cluster Interconnect

− Internal PureData Management Network

Large Rack

16

PureData


Cluster Elasticity Predefined cluster deployment patterns to handle

different transaction requirements

PureData system supports multiple pureScale clusters in a single system– Up to 3 pureScale clusters for the small configuration– Up to 6 for medium and 12 clusters for large

Shrinking of clusters is currently not supported (e.g. shrinking from 6 to 4 nodes)

Up to 10 DBs in a cluster.

* CFs and DB2 nodes are co-located and therefor cannot grow to a 4 node cluster.

cluster Size

# of Compute

Nodes

CFs DB2 Nodes# CPU

Cores# CPU

Cores

LARGE 6 2 32 4 64

MEDIUM 4 2 32 2 32SMALL 2 2* 16 2* 16

17

PureData


PureData for Transactions: Disaster Recovery

• Disaster Recovery is provided at the Database Level • Replication can be from one source to multiple targets• Source and target machines are not required to be identical• Data synchronization is maintained for an entire database or subset of objects• Data Synchronization Server (Q-replication or CDC) is running on a separate

machine

Data Synchronizationvia

Q-Replication or CDCData Synchronization Server

OR

Any server running DB2

PureData forTransactions

PureData forTransactions


Simple System Management

19

PureData


Patterns and Deployment

Pattern: pre-defined configuration that can be easily deployed

Cluster Patterns− Predefined database cluster architecture and configuration− Automatically creates, configures, and deploys

a cluster topology with built-in redundancy and high performance

− Operating system + all required software components

− Fully integrated, configured, and tunedfor a particular purpose

Database Patterns Configuration and best practices applied to a

database for a specific context Built-in pattern: Transactional (OLTP)

DB2 pureScale cluster options

20

PureData


Simplified and Integrated System Management

Single console to manage all resources and work running on the system– Consistent console across

IBM PureSystems

Role-based securityand tasks – Management– Monitoring – Maintenance

Easy integration with broader enterprise monitoring tools and processes

Two main views– System Console– Workload Console

IBM PureData System for Transactions IBM PureData System for Transactions

IBM PureData System for Transactions – Monitoring – Database IBM PureData System for Transactions – Monitoring – Database

21

PureData


System Console

System configuration Hardware monitoring

− down to a component level Console authentication

users and groups Maintenance functions

21

22

PureData


Workload Console

Managing DB2 pureScale clusters & databases

23

PureData


11

22

33

44

Creating a database cluster

24

PureData


11

22

33

44

Creating a database within the cluster

25

PureData


DeployDeploy Softwareapplication

HA Proxy

Caching Server Database

Application Server

HA Proxy

Application Server

Caching Server

Database

Simplified deployment across systems

Do you require High Availability

deployment ?

Do you require High Availability

deployment ?


DB2 pureScale for High Availability and Performance

27

PureData


Software ArchitectureDriving principles for the design of PureData System for Transactions Extreme Capacity

− Designed for OLTP− Grow easily with your business− Move from small to medium to large

Application Transparency− Avoid the risk and cost of changing applications

because of the database topology− Automatic Work Load Balancing, Automatic Client

Reroute, and Client Affinity

Continuous Availability− Maintain service across unplanned events− Maintain service across planned events using

maintenance

28

PureData


Technology Overview

DB2 engine runs on several host computers• Co-operate with each other to provide coherent access to the database

from any DB2 node

Data sharing architecture • Shared access to database• DB2 nodes write to their own logs• Logs accessible from another host (used during recovery)

Cluster caching facility (CF) from STG• Efficient global locking and buffer management• Synchronous duplexing to secondary ensures availability

Low latency, high speed interconnect• Special optimizations provide significant advantages on RDMA-capable

interconnects (10 Gigabit Ethernet (RoCE) or InfiniBand)

Clients connect anywhere…… see single database

• Clients connect into any DB2 node• Automatic load balancing and client reroute may change

underlying physical DB2 node to which client is connected

Integrated cluster services• Failure detection, recovery automation, cluster file system• In partnership with STG (GPFS,RSCT) and Tivoli (SA MP)

Leverages IBM’s System z Sysplex Experience and Know-How

Single Database View

DB2 DB2 DB2

Shared Storage Access

PrimaryCF

SecondaryCF

Log Log Log Log

Database

DB2

DB2 Node DB2 Node DB2 Node DB2 Node

CSCS CS

CS CS

CS

29

PureData


Scale with Ease

Without changing applications Efficient coherency protocols

designed to scale withoutapplication change

Applications automatically andtransparently workload balancedacross DB2 nodes

Without administrativecomplexity No data redistribution required

Single Database View

DB2 DB2

Shared Storage Access

PrimaryCF

SecondaryCF

Log Log Log Log

Database

DB2

DB2 Node DB2 Node DB2 Node

DB2

DB2 Node

30

PureData


Advantages of RDMA – An Example

RDMA over low latency fabric• Direct memory access• Enables short response time :

~10-15 microseconds Silent Invalidation

• Informs DB2 nodes of page updates, requiring no CPU cycles on those nodes

• No interrupt, No IP Socket Calls, No context switching, or other message processing

• Increasingly important as cluster growsHot pages available without disk I/O from GBP memory

• RDMA and dedicated threads enable read page operations in ~10s of microseconds

Node 1

Bufferpool

Node 2

Bufferpool

501

501

Node 3

Bufferpool

501

Node 4

Bufferpool

501

501

CF

GBP

GLM

501

Node 1 requests lock on page 501

Lock granted!

501

501

Silent Invalidation

31

PureData


Online Recovery

DB2 pureScale design point is to maximize availability during failure recovery processing

When a DB2 node fails, only in-flight data remains locked until node recovery completes In-flight = data being updated on the

failed node at the time it failed

Target time to row availability <20 seconds

32

PureData


Client Connectivity and Workload Balancing

Run-time load information used to automatically balance load across the cluster − Shares design with system z Sysplex− Load information of all DB2 nodes kept on each node− Piggy-backed to clients regularly− Used to route next connection (or optionally next transaction) to least loaded DB2 node− Routing occurs automatically (transparent to application)

Failover− Load of failed DB2 node evenly distributed to surviving nodes automatically

Fallback− Once the failed DB2 node is back online, fallback does the reverse


Security Model

34

PureData


Authentication

User authentication and group membership External (LDAP) – preferred

− Enforce password rules and policies− Centralized account management− Requires an external LDAP server (e.g. MS Active Directory)

− Internal (OS)− Functional IDs exempt

from policies Mix and match

35

PureData


System-level Security

Access to System Console and Workload Console Separation of duties

− Controls access to specific console objects, properties, and actions− Workload and database configuration management− Administration of resources: Workloads, cloud groups, hardware, audit, security− Assign “Read-only” or “Full Permission” to users or groups

System Console System Users, User Groups

36

PureData


Instance-level Security Controls what users/groups have access to instance and databases within Grant instance access to console users via Workload Console

− Menu Databases DB2 pureScale instances select an instance− cluster owner user (SYSADM) is created during instance deployment− “Access granted to” field: defines who can manage the instance− “Instance available to” field: defines who can deploy database in that instance

Create more users via Workload Console− Menu Databases DB2 pureScale instances select an instance Manage− Creates users and groups in OS or LDAP− Can grant SSH access to Compute Nodes

36

37

PureData


Database-level Security Instance owner has SECADM and DBADM authorities.

− SECADM is revoked from instance owner and only default DB user has this right after database is deployed.

Create users and groups via Workload Console− Database Databases <select database> Manage− Created in OS or LDAP− Can grant SSH access to compute nodes

Grant/revoke permissions as usual in DB2 − DB2 CLP or IBM Data Studio

37

38

PureData


Thanks!

Questions?Please feel free to reach out to me anytime, if questions come to mind:

Fraser McArthur ([email protected])

mailto:[email protected]

Documents

PureData for Transactions Overview - public.dhe.ibm.compublic.dhe.ibm.com/software/dw/puresystems/tech... · © 2013 IBM Corporation PureData for Transactions Overview Fraser McArthur