Embed Size (px)
Citation preview
1
May 14, 2009 • 9:45 a.m. – 10:45 a.m.Platform: all
Curt CotnerIBM FellowVP & CTO for Database Servers
Session: H09Optimizing Your WebSphere
Applications with Data Studio
2
2
Introducing Integrated Data Management from IBM Optim™
• Grow the business, without growing costs
• Develop and deploy business critical applications faster
• Mitigate compliance risks with model-driven data governance
• Prevent runaway infrastructure spending• Improve performance – of teams,
databases, applications, and business units
An integrated, modular environment to manage enterprise application data, and optimize data-driven applications, from requirements to retirement
IBM has responded to these challenges with the concept of integrated data management, and the portfolio of integrated data management solutions from IBM Optim.
Optim offers capabilities to manage business critical data, and to optimize the performance of data driven applications, across their lifecycle. Optim supports each phase of the lifecycle with robust offerings for data-centric tasks and roles, and has begun to build out the integration that will facilitate consistent, effective data management from requirements to retirement. It offers IBM clients the ability to grow the business, without growing the associated cost base. <Note: the individual bullets that follow describe the key elements of the message model.>
3
3
Data Studio has evolved into Integration Data Management• Key Objectives Remain
• Providing end-to-end data lifecycle management• Spanning heterogeneous databases• Improving productivity and cross-organizational
collaboration • Integrating via common user interfaces, common
components and services, and shared policies, models, and metadata
4
4
InfoSphere Data Architect
Data Studio DeveloperOptimization Expert for z/OS
Data Studio Offerings – January 2009
DB2 Performance Expert and Extended Insight
Data Studio pureQuery Runtime
Data Studio AdministratorDB2 High Performance Unload
Develop
Design
Deploy
Optimize
Operate
PoliciesModels
Metadata
Data Studiono-fee offering
5
5
Optim Studio DeveloperOptim Workload Tuner for
z/OS
Optim Offerings – June 2009
Optim pureQuery
Develop
Design
Deploy
Optimize
Operate
PoliciesModels
Metadata
Optim Query Tuner
InfoSphere Data Architect
DB2 Performance Expert and Extended Insight
Optim Database AdministratorOptim High Performance Unload
Optim Data Studiono-fee offering
6
6
Optimizing Your WebSphere Applications
7
7
What’s so Great About DB2 Accounting for CICS Apps?
z/OS LPARCICS AOR1Txn1
- Pgm1- Pgm2
CICS AOR2TxnA
- PgmX- PgmY
DB2PROD
CICS AOR3Txn1
- Pgm1- Pgm2
App CPU PLANTxn1 2.1 TN1PLNTxnA 8.3 TNAPLN
DB2 Accounting for CICS apps allows you to study performance data from many perspectives:• By transaction (PLAN name)• By program (package level accounting)• By address space (AOR name)• By end user ID (CICS thread reuse)
This flexibility makes it very easy to isolate performance problems, perform capacity planning exercises, analyze program changes for performance regression, compare one user’s resource usage to another’s, etc.
8
8
JDBC Performance Reporting and Problem Determination – Before pureQuery
Application ServerDB2 or IDS
A1
A2
A5
A3
A6
A4
USER1
USER1
USER1
User CPU PACKAGEUSER1 2.1 JDBCUSER1 8.3 JDBCUSER1 22.0 JDBC
What is visible to the DBA?- IP address of WAS app server- Connection pooling userid for WAS- app is running JDBC or CLI
What is not known by the DBA?- which app is running?- which developer wrote the app?- what other SQL does this app issue?- when was the app last changed?- how has CPU changed over time?- etc.
Data A
ccess Logic
Persistence Layer
DB2 Java D
river
EJB Query Language
9
9
What’s so Great About Optim pureQuery Accounting for WebSphere Applications?
z/OS LPAR
CICS AOR2TxnA (PLANA)
- PgmX- PgmY
App CPU TxnA 2.1 TxnB 8.3
Data Studio and pureQuery provide the same granularity for reporting WebSphere’s DB2 resources that we have with CICS:• By transaction (Set Client Application name )• By class name (program - package level accounting)• By address space (IP address)• By end user ID (DB2 trusted context and DB2 Roles)
This flexibility makes it very easy to isolate performance problems, perform capacity planning exercises, analyze program changes for performance regression, compare one user’s resource usage to another’s, etc.
Unix or Windows
WAS 21.22.3.4TxnA (Set Client App=TxnA)
- ClassX- ClassY
10
10
Simplifying Problem Determination Scenario
Application Developer• Available for each db access
• SQL text generated• Access path• Cost estimates• Estimated response time• Elapsed & CPU time• Data transfer (getpages)• Tuning advice
Database Administrator
• Available for each SQL• Application name• Java class name• Java method name• Java object name• Source code line number• Source code context• J-LinQ transaction name• Last compile timestamp
Java Profiling
pureQuery
DRDA Extentions
11
11
Using pureQuery to Foster Collaboration and Produce Enterprise-ready Apps
Application Server
Catalog data for SQL
ApplicationMeta data
DB2 or IDSProd
A4
A1
A1
A6
A6
A2
A2
A3
A3
A4
A4
A5
A5
A1
A4
A5
PerformanceData
Warehouse
Application Developer Database
Administrator
A1
A6A2 A3
A4 A5
Quickly compare unit test
perf results to production
Use pureQuery app metadataas a way to communicate in terms familiar to both DBA and developer
ApplicationMeta data
DB2 or IDSDev System
A1
A6
A2 A3
A4 A5A1
A4
A5
12
12
Java Persistence Technologies with pureQueryJPA API pureQuery API
JPA Runtime
pureQuery Runtime
JDBC w/pureQuery
IBMDatabase
pureQuery Metadata, Manageability
SpringiBatisJDBC
SQLJ
High Speed API
13
13
Client OptimizationImprove Java data access performance for DB2 –without changing a line of code
• Captures SQL for Java applications• Custom-developed, framework-based, or packaged applications
• Bind the SQL for static execution without changing a line of code• New bind tooling included
• Delivers static SQL execution value to existing DB2 applications• Making response time predictable and stable by locking in the SQL access path
pre-execution, rather than re-computing at access time• Limiting user access to tables by granting execute privileges on the query
packages rather than access privileges on the table• Aiding forecasting accuracy and capacity planning by capturing additional
workload information based on package statistics • Drive down CPU cycles to increase overall capability
• Choose between dynamic or static execution at deployment time, rather than development time
14
14
Data Studio pureQuery Runtime for z/OS
• In-house testing shows double-digit reduction in CPU costs over dynamic JDBC
•
•
• IRWW – an OLTP workload, Type 4 driver• Cache hit ratio between 70 and 85%• 15% - 25% reduction on CPU per txn over dynamic JDBC
274360
420 446485
524
0
100
200
300
400
500
Nor
mal
ized
Thr
ough
put (
ITR)
EJB
2
JPA
JDBC
pQ M
etho
d Dy
nam
icCl
ient
Opt
imiz
n St
atic
pQ M
etho
d St
atic
Normalized Throughput by API for JDBC Type 4 Driver
-35%
-14%
6%15%
25%
-50%
% in
crea
se/re
duct
ion
in C
PU p
ertr
ansn
com
pare
d to
JD
BC
EJB
2
JPA
pQ M
etho
d Dy
nam
icCl
ient
Opt
. Sta
ticpQ
Met
hod S
tatic
% increase/reduction in CPU per transaction compared to JDBC using Type 4 driver
15
15
OpenJPA and Hibernate --SQL Query Generation
JPA Query
Select emp_obj(),dept_obj()
SQL
Select * from EMPWHERE …
Select * from DEPTWHERE …
JPAquery transform
• Hibernate and OpenJPA often rewrite queries• No database statistics are used – entirely heuristic!!!• Can often result in poorly performing queries
16
16
pureQuery -- More Visibility, Productivity, and Control of Application SQL• Capture SQL• Share, review, and optimize SQL• Revise/optimize SQL and validate equivalency
without changing the application• Bind for static execution to lock in service level or
run dynamically • Restrict SQL to eliminate SQL injection
Capture Review Optimize Revise Restrict
Manage static SQL deployment: This release builds out additional capabilities to enhance developer and DBA collaboration and manage static SQL execution
Empower developers or DBAs to customize captured SQL before binding Select which SQL statements are boundDelete SQL statements from packages Replace existing SQL with an equivalent, and potentially more optimal, SQL without modifying the program source code
Enable developers to give DBAs deployment-ready files for package binding Improve feedback on bind errors including SQL statements within the package that caused bind to failSimplify bind file development (automate SQL file discovery within a single project for bindprops) and manage binds across jar, war, and ear files used for deploymentAvoid unnecessary binds when redeploying a jar when only a subset of contained applications has changed
17
17
Visualize execution metrics
Execute, tune, share, trace, explore SQL
Replace SQL without changing the application
Position in Database Explorer
Visualize application SQL
18
18
OpenJPA and Hibernate --Batch Queries
JPA Query
new dept_obj …new emp_obj …new dept_obj …new emp_obj …
SQL
INSERT INTO DEPT …INSERT INTO EMP …INSERT INTO DEPT …INSERT INTO EMP …
JPAquery rewrite
• Hibernate and OpenJPA can “batch” queries to reduce network traffic• Batches must contain executions of a single prepared statement• Referential integrity constraints can change batch size:
• 2 network trips without RI (one for EMP, one for DEPT)• 4 network trips if RI disables batching
• pureQuery can convert the above example to a single network trip, regardless of whether RI is used or not…
19
19
Have You Heard of SQL Injection?
From: http://www.itpro.co.uk/605127/timeline-outbreak-the-rise-of-the-sql-infection.
20
20
Toughest issue for Web applications –Problem diagnosis and resolution
Web BrowserUsers Web
Server
Application Server
DB2Server
Business Logic
Data A
ccess Logic
Persistence Layer
DB2 Java D
river
JDBCPackage
EJB Query Language
21
21
Customer Job Roles –A Barrier to a “Holistic View”
Application Server
DBServer
Data A
ccess Logic
Persistence Layer
DB
Java Driver
JDBCPackage
EJB Query Language
WebSphere
Connection
Pool
BusinessLogic
13
54
2
ApplicationDeveloper
SystemProgrammer DBA
NetworkAdmin
22
22
DB2 Performance Expert and Extended Insight
• Enables early and rapid problem detection to prevent impact to production systems
• Identify resource shortages across CPU, memory, and file systems, locking conflicts and deadlocks, and data skew
• Increase ability to meet service level agreements
• Provides optimization and tuning recommendations
• Supports trend analysis and growth planning
• Maintains and analyzes performance warehouse
• Supports a variety of database workloads • Includes online transaction processing, data
warehouse, and enterprise content management
• Monitors Workload Management environment
• Visualizes the end-to-end response time • See where Java database workloads,
transactions, and SQL requests are spending their time across the database client, the application server, and the network
Provides in-depth database monitoring, problem isolation, and trend analysis for DB2 for
Linux, UNIX, and Windows databases. Add Extended
Insight Feature for visibility into where Java database workloads, transactions, and SQL requests
are spending their time
Modern Java based composite application environments have created huge benefits from their flexibility in application development and deployment. However, these composite applications have also created huge challenges for DBAs in understanding and managing overall database performance within those environments. Database requests are still sent from the application to the database, and database responses are still sent back to the application. But with an increasing number of additional layers between the actual line of application code and the underlying database server, the DBA has been encountering an increasingly difficult barrier in working with the application programmer to find and fix database application performance problems.
DB2 Performance Expert Extended Insight Feature enhances DB2 Performance Expert to solve these problems by providing the DBA with an end-to-end database monitoring solution for a complete view of database performance across the database client, the application server, the database server, and the network between them.
With the Extended Insight Feature, DBAs get a complete view of the overall database system performance from the same perspective as the database application. DBAs can now: •Monitor key performance indicators to proactively detect negative trends in:
•Response times for database APIs •Network congestion between the application and the database, or•Other factors key to sustaining service level agreements
•See where workloads, transactions, and SQL requests originate in the application, and how the time spent is spread across the database client, the application server, the database server, and the network between them:
•Use graphical displays to pinpoint problems•Filter by application, database, user, or other criteria
•Drill down into relevant detail to isolate problems quicklyWhen used with WebSphere Application Server, the DB2 Performance Expert Extended Insight Feature gathers additional performance indicators, including WebSphere Application Server connection pool wait times and other vital database-related performance statistics.
With new visibility and monitoring of the complete database request and response performance, DBAs are able to either reduce the time needed to isolate the sources of database application performance issues from days to minutes, or when the problem liesoutside of the database system, to eliminate the database as the potential source of database application performance issues, freeing themselves to focus on business value creation.
Thus DBAs are able to manage to service level agreements more easily and effectively.
23
23
How do we plan to help?
24
24
Scenario
It seems that the first application server has a problem. Double-click to drill-down.
In this situation, all applications are equally affected, and the problem seems not to be in the data server.
25
25
Scenario - continued
Double-click to drill-down and display detail information.
Most of the time is spent for „WAS connection pool wait“ time.
26
26
Scenario – continued
5 second wait time indicates that the maximum number of allowed connections is not sufficient…
… which becomes also evident when comparing the parameters and metrics of this client with other clients.
27
27
Optim Database Performance Manager future directions -- Associate SQL with Java Source
Heat Chart DashboardAlerts SLAs In-flight analysis
Database: Accounting
435
221
1323
occurrences
32322.3
11.3
123.32
sort time
PROC
SYSIBM
SAP3schema
23.3
323.4
132.13
E2E elapsed
32.1DELETE FROM ACCOUNT WHERE AID = 3…
32.1SELECT SALARY FROM PAYMENT …
1.303SELECT TIME FROM UNIVERSE
Statement text phys. I/O
TOP 3 currently running SQL Statements
TOP byDS elapsedDS CPU timePhysical I/OSort time
- +
- +
- +
SELECT TIME FROM UNIVERSEStmt text Analyze
Application
West.OLBankpackage
Transfer()method
Online bankingapplication [email protected] contact
Accountclass
KARNDS user IDTPKARN.de.ibm.comClient IP addr / hostname
TPKARNClient workstation name
314source line
N/AClient accountingJawaw.exeClient application name
Client user ID KARN
Time distribution
Force applicationStop SQL
sorting
18.456Query cost estimatesBuffer Pools
43.4%Data – hit ratio (%)
3214Index – physical reads / min54.2%Index – hit ratio (%)4323Data – physical reads / min
Resource usage
Statement information X
DS Proc
USER CPU SYSTEM CPU
Unacc wait
DS sorting
132.13 sec Current239.40 seclast day15.60 seclast week
Statement elapsed time
Top-level overview
28
Support for Oracle
29
29
Design, Develop, and Deploy for OracleUnified solution across DB2, Informix, and Oracle• Design – InfoSphere Data Architect
• Logical models, physical models, privilege models, privacy models
• Platform-specific for physical objects• Develop – Optim Studio Developer
Generate data access layers• Develop and debug procedures, SQL,
functions including PL/SQL• Test - Optim Test Data Manager and
Data Privacy solutions• Invoke Optim TDM from Developer• Use relationship and privacy
definitions from InfoSphere Data Architect
• Create right-sized, fictionalized, production-like test databases
• Deploy – Optim Studio Developer • Create/manage database objects• Copy/paste objects between DB2 and
Oracle
Load Oracle catalog
Edit database objects
View SQL and execution results
Take context-sensitive actions
Coming 2009
30
Oracle Object Management support in Data Studio
•Create/Alter/Drop for all objects•Tables, Synonyms, Sequences•Functions•Views/materialized views
Physical objectsPerformance objects
Integrity objects
Procedures and Functions
User Defined Types
EventsSpace•Create/Alter/Drop•Tablespace, Extents, Free Lists, logging•LOB Attributes•Buffer pools
•Create/Alter/Drop •Partitions (Range, Hash, List)•Indexes
•Create/Alter/Drop•Constraints (primary, unique, check, foreign)
•Triggers•Before/After/Foreachtypes•Trigger events
•Create/Alter/Drop
•Table Types•Object Types•Array Types
Strengthening Oracle Support
Strengthening Oracle Support
31
31
Advanced heterogeneous support
IBM ORACLE
Physical Data Modeling
Logical Data Modeling
Oracle Support• Visualize• Design Privileges • Storage and Data Partition• Advanced Code Generation• Analyze Impact• Validate
IBM Data Studio
32
32
Visualize Oracle Data Sources• High fidelity display of the Catalog Information• Load on Demand technology
• Instantaneous connection• Fast retrievals
• Enable Physical Data Model transformation
33
33
Managing Oracle Tables
Tree-Based Representation
Object Properties
Editor
SQL and Results of the
ExecutionContext-Sensitive Actions
34
34
Oracle Privileges Support
• Physical Model enables Design capability• Grant appropriate privileges and roles to users
• More detailed display allows finer-grained control
35
35
Oracle Storage
• Storage properties display• Ability to design Table Spaces
36
36
Data Partition
• Table and Materialized View support• Range partition• List partition• Hash partition• Composite partition
37
37
PL/SQL Development• Integrated Query Editor support
• Content Assist• Parser support (2009) with Error reporting
38
38
PL/SQL Deployment
• Execution Configuration• Separation of concerns
• Better decouple configuration from definition and implementation Deployment and Debugging
• Runtime Target initialization• Authorization configuration
39
39
Data & Object Movement• Value Proposition –
• Provide for the copying of database objects and data between homogeneous and heterogeneous databases within Data Studio
• Key Features• Copy objects at various levels –
complete databases to a fixed number of rows from a single table
• Action performed in Data Source Explorer – Copy/Paste and Drag/Drop
• Can automatically copy rows from related tables using:
• RI in database• Data Architect model• Optim application models• Data Relationship Analyzer
• Can optionally annonymize the rows using Optim Test Database Manager
40
40
Common Connection Repository• Enhancing the value propositions on
Team support• Centralized connection properties for sharing
between DBA and Developers• Improve usability and up-and-running
scenarios• Give controls to DBAs on connection
properties settings• Eliminates the need to configure each
database server on each client desktop• “push down” of client properties to allow DBAs
to control and override application behaviors
• Key Features• Integrated solution to Eclipse Data Source
Explorer• Integration with upcoming Web DBA tooling • Create or connect to Connection Repository• Connect to database using existing definitions• Create new definition• Logical grouping of connection definitions
41
41
Where to get IBM Data Studio ?
• IBM Data Studio• www.ibm.com/software/data/studio
• FAQs / Tutorials • Downloads• Forum / Blogs• Join the IBM Data Studio user
community
42
42
43
43
Curt CotnerIBM
Session: H09
Optimizing Your WebSphere Applications with Data Studio
