Building and Managing Efficient data access to · PDF fileBuilding and Managing Efficient data access to DB2 ... Impede capacity planning ... Table 1, operation 4 DBMS

September 16, 2010

Building and Managing Efficient data access to DB2

Vijay Bommireddipalli, [email protected] Architect, Optim

Information Management

© 2010 IBM Corporation2

Disclaimer

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF IBM SOFTWARE.



Open Source Persistence

Engine

JPA Persistence Engine

JDBC API

Hibernate, iBATIS, EclipseLink,...

JPA for WebSphere, Apache OpenJPA

DAOAPI

DataAccess Object

pureQueryPlain JDBCsMash

PHP, Ruby

Application on-ramps to DB2

JCC driver

JDBC

.Net Applications

.Net applications

JPA APIOpen Source

Persistence API Web API

ADO .Net

DB2

IBM Information Management


Layered Application Stack Challenges

� Simplify development, but …� Challenge problem resolution � Impact performance� Obscure impact analysis� Impede capacity planning

� Layers obscure linkages� Where is the problem?� Which application is running?� Which application does this SQL

belong to?

Application Server Database Server

Data A

ccess Logic

Persistence Layer

DB

Java Driver

EJB

Query Language

Connection

Pool

BusinessLogic

1

2

3

Network

A

A

A



Layered Application Stack Challenges

� Simplify development, but …� Challenge problem resolution � Impact performance� Obscure impact analysis� Impede capacity planning

� Layers obscure linkages� Where is the problem?� Which application is running?� Which application does this SQL

belong to?

Application Server Database Server

Data A

ccess Logic

Persistence Layer

DB

Java Driver

EJB

Query Language

Connection

Pool

BusinessLogic

1

2

3

Network

A

A

A

Application

Developer

System

s A

dministrator

Netw

ork A

dministrator

Database

Adm

inistrator



� Resource utilization across application and databas e infrastructure

� DBAs have little to no visibility of application SQL before deployment, no opportunity for review and optimization

� Problem isolation takes days due to inability to trace SQL to Java application and source code

� Constantly increasing application workload taxes existing systems – need to fit more work into existing systems

� SQL injection represents an increasing risk to data security

Data Access –Two Views of the World

DeveloperDBA

� No insight into data access performance

� Databases are a different skills domain

� Not a productive development environment

� No single environment that spans Java application and database development

� Problem isolation and resolution

mashuphttp

JSON

JSP

XML

JDBC

SQL

Spring

StoredProcedures

QoS goals Runstats

Response Time! REORG

Partition strategy



Can use individual

Statements

Using

Dynamic or

static SQL

getCustomers()getCustomersbyZip()getAccountDetails()getAccountReport()

ClientApplications

(business logic)

Static SQL

Packages

Database access development options -

getCustomers()getCustomersbyZip()getAccountDetails()getAccountReport()

Application Version 1

Uses individual SQL

Application Version 2

Uses SP Calls Db access

Using

Stored Procs

Call can be

Dynamic or

Static SQL

Call Individual SQL statements

Call Stored Procedures (SQL SPs, Java SPs),Stored Procedures – Can be SQL bodied or Java

Stored Procedures



Comparing programming models

A. Application SQL onlyDynamic SQL

B. SQL Stored ProcedureDynamic CALL, Static Procedure

C. Application SQL onlyStatic SQL

D. SQL Stored ProcedureStatic CALL, Static Procedure

SELECT C1 FROM T1

ApplicationDB

CALL P1

ApplicationDB

SELECT C1 FROM T1

SELECT C1 FROM T1

Application

SELECT C1 FROM T1

DB

CALL P1

Application

DB

SELECT C1 FROM T1

CALL P1

Information Management – IBM Data Studio


SQL Execution - Dynamic vs. Static

Dynamic SQLCheck auth for package / plan

Parse SQL StatementParse SQL Statement

Check Table / View AuthCheck Table / View Auth

Calculate access pathCalculate access path

Store access path in a temporary package

Store access path in a temporary package

Execute SQL statementExecute SQL statement

Extract access path from catalog or package cacheExtract access path from catalog or package cache

Execute SQL statementsExecute SQL statements

Static SQLCheck auth for package / plan



SQL Execution AdvantagesDynamic vs. Static

Feature Dynamic SQL(pureQuery, JDBC)

Static SQL(pureQuery, SQLJ)

Performance Can approach static SQL performance with help from dynamic SQL cache. Cache misses are costly

All SQL parsing, catalog access, done at BIND time. Fully optimized during execution.

Access path reliability

Unpredictable – Any prepare can get a new access path as statistics or host variables change

Guaranteed – locked in at BIND time All SQL available ahead of time for analysis by EXPLAIN.

Authorization Privileges handled at object level. All users or groups must have direct table privileges – Security exposure, and administrative burden

Privileges are package based. Only administrator needs table access. Users/Groups have execute authority. Prevent non-authorized SQL execution.

Monitoring, Problem determination

Database View is of the JDBC or CLI package – No easy distinction of where any SQL statement came from.

Package View of applications makes it simple to track back to the SQL statement location in the application

Capacity planning, Forecasting

Difficult to summarize performance data at program level.

Package Level Accounting gives program view of workload to aid accurate forecasting.

Tracking dependent objects

No record of which objects are referenced by a compiled SQL statement

Object dependencies registered in database catalog



Have You Heard of SQL Injection?



pureQuery ClientOptimization Static

pureQuery API Static(Method Style)

97%115%

0%

20%

40%

60%

80%

100%

120%

ITR

: % d

iffer

ence

from

JD

BC

90% Package Cache hit ratio: DB2 LUW Database ITR comparison to JDBC

Increased throughput as compared to JDBC with 90% Package Cache hit ratio

In comparison with JDBC, Static APIs doubled the throughput (ITR*) of the DB2 LUW database server.



pureQuery ClientOptimization Static

pureQuery API Static(Method Style)

60% 66%

0%

20%

40%

60%

80%

100%

ITR

: % d

iffer

ence

from

JD

BC

95% Package Cache hit ratio: DB2 LUW Database ITR comparison with JDBC

Increased throughput as compared to JDBC with 95% Package Cache hit ratio

Compared to JDBC, Static APIs increased the throughput (ITR*) ofthe DB2 LUW database server by 60 and 66% respectively



Performance

� Using IRWW SQLJ and JDBC workloads, CL.1 CPU drops by 19.22%, while CL.2 CPU drops by 20.51%. Throughput increases by 46%

SQLJ and JDBC - dynamic vs. static SQL

0

0.0002

0.0004

0.0006

0.0008

0.001

0.0012

0.0014

0.0016

0.0018

0.002

Dynamic CL1 Static CL1 Dynamic CL2 Static CL2

Mic

rose

cond

s Dynamic CL1

Static CL1

Dynamic CL2

Static CL2



Static CALL to native SQL SP vs. dynamic CALL

� Results show an 11% reduction in CL.2 CPU time, and 9% reduction in CL.1 CPU time. Elapsed times dropped by roughly 5%.



Stored Procedures and DB2

� SQL � DB2 for LUW Version 8.2, 9.1, 9.5, 9.7� DB2 for iSeries V5R3, V5R4

� SQL External� DB2 for z/OS Version 8, 9

� SQL Native � DB2 for z/OS Version 9

� Java (JDBC or SQLJ or pureQuery)� DB2 for LUW Version 8.2, 9.1, 9.5, 9.7� DB2 for iSeries V5R3, V5R4� DB2 for z/OS Version 8, 9

� PL/SQL� DB2 for LUW Version 9.7�Oracle V10, V11



Stored procedures in DB2 9 for z/OS - SQL

� External SQL procedures:• Pre- DB2 9 SQL stored procedures• SQL processor converts SQL statements into a C program• Build utility DSNTPSMP, customizable • Runs in a WLM environment• Debug with Unified Debugger

� Native SQL procedures:• SQL procedure logic runs in the DBM1 address space• Extensive support for versioning• Native SQL procedures support zIIP Processors when invoked

over DDF• Debug with Unified Debugger



SQL Execution mode in Java Stored Procedures

� Dynamic :• Uses pureQuery or JDBC APIs

� Static :• Uses pureQuery or SQLJ

� Debug both with Unified Debugger



Comparing programming models

A. Application SQL onlyDynamic SQL

B. SQL Stored ProcedureDynamic CALL, Static Procedure

C. Application SQL onlyStatic SQL

D. SQL Stored ProcedureStatic CALL, Static Procedure

SELECT C1 FROM T1

ApplicationDB

CALL P1

ApplicationDB

SELECT C1 FROM T1

SELECT C1 FROM T1

Application

SELECT C1 FROM T1

DB

CALL P1

Application

DB

SELECT C1 FROM T1

CALL P1

High QoSHigh QoS

C & D Higher performance, access path reliability, improved authorization, monitoring, problem determination, capacity planning, and dependent object tracking.

B & D Use SPs when several SQL statements work together with

minimal flow logic



� Review SQL coming from applications

High cost of literals

� Consolidate SQL by replacing literals with parameter markers� Optimize database resources (e.g. dynamic statement cache) usage



Table 1, operation 5




Table 1 , operation 4


Using JDBC batching

Users

Application Servers

Network

DBMS

Batching – Multiple operations all execute as one batch – minimize chattiness on the network

pStmt.setString(2,"Accounting");

pStmt.setString(3,"A00");

pStmt.addBatch();

int [] numUpdates=prepStmt.executeBatch();









SQLJ – using Heterogenous batching

Users

Application Servers

Network

DBMSTable 1, operation 4

Heterogenous Batching – Multiple operations across different tables all execute as one batch



Java Data Access Spectrum – balancing productivity and control

Managed objects Object-relational mapping

Spring templates

Full SQL control

Code all your SQL

Use SQL templates, inline only

Complex OR mapping and persistence management, but loss of controls

Adds container management option

JDBC / SQLJ

iBATIS

Hibernate

OpenJPA (EJB3)

Add basic OR mapping and annotated-method style

pQ DAO


© 2007 IBM Corporation24 IBM Data Studio pureQuery For DBAs and Application Developers (v2.1) - Part 1 24242424

SQLJSQLJ

####sqlsqlsqlsql [con] {SELECT [con] {SELECT [con] {SELECT [con] {SELECT firstnmefirstnmefirstnmefirstnme INTO :INTO :INTO :INTO :empNameempNameempNameempName FROM employee WHERE FROM employee WHERE FROM employee WHERE FROM employee WHERE empnoempnoempnoempno = := := := :empnoempnoempnoempno};};};};

JDBC

SringSringSringSring sqlStatementsqlStatementsqlStatementsqlStatement = = = = ““““SELECT SELECT SELECT SELECT firstnmefirstnmefirstnmefirstnme FROM employee WHERE FROM employee WHERE FROM employee WHERE FROM employee WHERE empnoempnoempnoempno = ?= ?= ?= ?””””;;;;

java.sql.PreparedStatementjava.sql.PreparedStatementjava.sql.PreparedStatementjava.sql.PreparedStatement pspspsps = = = = con.PrepareStatement(sqlStatementcon.PrepareStatement(sqlStatementcon.PrepareStatement(sqlStatementcon.PrepareStatement(sqlStatement););););

ps.setString(1, ps.setString(1, ps.setString(1, ps.setString(1, empnoempnoempnoempno););););

java.sql.ResultSetjava.sql.ResultSetjava.sql.ResultSetjava.sql.ResultSet reader = reader = reader = reader = ps.executeQueryps.executeQueryps.executeQueryps.executeQuery();();();();

reader.nextreader.nextreader.nextreader.next();();();();

empNameempNameempNameempName = reader.getString(1);= reader.getString(1);= reader.getString(1);= reader.getString(1);

Reader.closeReader.closeReader.closeReader.close(); (); (); ();

Non-Java syntax, but optimal accessTedious and long code

SQLJ Pros SQLJ Cons

Concise compared to JDBC Not a popular common programming model

Full control of SQL Many IDEs do not have SQLJ support

Most optimal access to the database (static) Deployment has additional steps

Heterogenous batching (reduce network traffic)

Cannot be used with other programming models

Better diagnostics



Data Access Objects – core J2EE pattern

Using J2EE Data Access Objects

� Methods for each database access

� Each method has only your parameters

� SQL can be in separate file – separate business logic from data access



What is Hibernate

� Open source Object-Relational Mapping (ORM) library for Java

� Simple object oriented database access

� No complex JDBC coding

� Data access though object query language and SQL

HibernateHibernateHibernate

Database

JNDI JDBC JTA

ApplicationApplicationApplication



HibernateHibernateHibernate

Database

JNDI JDBC JTA

ApplicationApplicationApplication

Hibernate

Developers code as above

Hibernate generates the following SQL



Goals for performance management

� Prevent problems before they impact the business�Get early warning of emergent problems� Isolate problems quickly

� Improve quality of service� Use expert advice for faster problem resolution� Stabilize application performance

� Reduce costs� Increase capacity of existing systems� Free up DBA time for value creation activities� Fix problems in development while costs and impact are low� Propagate performance expertise to avoid problems



Users

Application Servers

Network

Identify, diagnose, solve, and prevent performance problems

• Drill-down into problem detail and related context• Analyze captured SQL

2. Diagnose

• Alert of potential problems• Visual quick scan of complex environment

1. Identify

4. Prevent

• Use proven performance improvement techniques

3. Solve

DBMS & OS

• Use best practices to avoid problems in development



More Visibility, Productivity, and Control of Application SQL

� Capture � Capture the SQL during development

� Optimize� Review and share SQL� Optimize and replace bad SQL � Create approved SQL list

� Deploy � Configure execution properties� Optionally bind for static execution

Capture

Review Restrict

Configure Bind

Execute

Monitor

Optimize

Java Persistence Architecture



Simplifying Problem Determination Scenario

Application Developer

�Available for each db access�SQL text generated�Access path�Cost estimates�Estimated response time

Database Administrator

� Available for each SQL�Application name�Java class name�Java method name�Java object name



Which application has issued the problematic SQL?

Facilitate Problem Determination

� Use database package information to correlate SQL w ith Java or .NET applications

Diagnose



Unique Package Names Improves PD

DynamicJava SQL

StaticJava SQL

� Hard to identify unique Java applications

� Most dynamic Java applications use packages SYSLNx00

� Unique package names link SQL to Java code modules

Diagnose



Statically invoking Stored Procedures from Java

Native SQL stored procedure invoked Dynamically from Java application

Native SQL stored procedure invoked Statically from Java

application

Diagnose



Client Info. fields

� Voluntary – but very useful !� java.sql.Connection.setClientInfo -� setClientInfo("ApplicationName","OrderHistoryApp");� conn.setClientInfo("ClientUser", “Vijay Bommireddipalli"); � conn.setClientInfo("ClientHostname, “xyz");

Diagnose



Tip: Adding Java clientProgramName will set Correlation Id. Diagnose



Solve using any or all of the following -

� Stabilize performance and lock access plans using Static SQL

� Improve performance using JDBC batching or even better –Heterogeneous batching capability

� Improve performance by substituting literals with parameter markers

Solve



Preventing the problem: Proactively closing the loop

Improve usage of database

Execute application

Monitor

Improve databaseperformance

Application Developers

DBA

Prevent



Building high performance applications proactively

� Take advantage of optimized database access� Static SQL� Batching (JDBC or heterogeneous)� Avoid literals where possible� Avoid unnecessary re-prepares� Stored Procedures

� Add diagnostics where possible� Eg. Client info fields� ‘Side effects’ of static SQL

� Templates� Share templates of frequently used SQL statements

� Review process for SQL� Explain analysis� Tune early, Tune often

Prevent



Resources

� Optim solutions for accelerating Java database accesshttps://www.ibm.com/developerworks/offers/lp/demos/summary/im-optimsolutionsforjava.html

� IBM Optim Development Studiohttp://www-01.ibm.com/software/data/optim/development-studio/

� IBM Optim pureQuery Runtime web page http://www.ibm.com/software/data/studio/purequery/

� Articles and tutorials covering the Optim portfoliohttp://www.ibm.com/developerworks/spaces/optim



Documents

Building and Managing Efficient data access to · PDF fileBuilding and Managing Efficient data access to DB2 ... Impede capacity planning ... Table 1, operation 4 DBMS