© 2012 Progress Software Corporation. All rights reserved. 1 Click to edit Master title style Remove Performance Bottlenecks in Your SQL Connections -

© 2012 Progress Software Corporation. All rights reserved.1

Click to edit Master title styleRemove Performance Bottlenecks in Your SQL Connections - Data Connectivity Performance Best Practices

Steve Pittman, ArchitectOpenEdge SQL

PUG Challenge Americas – 2012 -


Disclaimer

The future features described in this presentation are under consideration by Progress Software and are not commitments for future products, technologies, or services.

Any forward looking statements represent our estimates only as of today and should not be relied upon as representing our estimates as of any subsequent date. While we may elect to update forward looking statements at some point in the future, we specifically disclaim any obligation to do so, even if our estimates change.


Agenda

Where did all that data come from? What can we do?

A systematic approach to performance

An OpenEdge perspective

Questions?


A Brief Audience Survey

# using ODBC– Tools – Crystal?

# using JDBC– Tools – DbVis?

# using Java Java appservers

– WebLogic?


Humans are information machines

Humans are now storing over 988 Exabytes of information (at the end of 2010)

15 petabytes of information generated every day

Scientific, Engineering, Financial, Social – all areas of our lives continue to generate and store the world’s data at an increasing rate.


Making this data useable

Performance trends in 2010-11 were centered around making this data more quickly accessible (and more useful).

Sorting, cataloging, querying, inserting…all these are required for making the data mean something – and it has to be fast!


Performance trends in 2012

Virtualization made efficiency a way of life

Concurrency will gain steam in 2012– Research on multi-core has been building– New tools by chip makers will help – Additional Language constructs and

compiler directives will come– Java application development frameworks– Application servers – WebLogic, JBoss

Larger buffers and algorithm changes will take place to fully utilize 64 bit processors.


How did we get here?

1982 2012

The advance of information technology (hardware and software) gives you a dizzying array of choices on how to store your data….

…and not everyone has the same opinion.

Multi-tenant db


A typical Enterprise App Architecture

Database

Buffers, I/O, etc..

Network efficiency

Database Settings

Application UI

Business Logic &Data Access Code

Driver


Time is of the essence

Large amounts of data require long processing times – small percentage gains result in large time savings

Performance tuning steps remain the same


So, what can we do to keep up?

Aggressively analyze all pieces of your application architecture.

Just follow the PANDDa.


Where Performance Bottlenecks Lie

P.A.N.D.Da

Platform

Application Code

Network

Driver

Database

Which element will give the most payback?


Platform

Consider native calls into the Platform and whether you can do them more efficiently.

Varying client platform performance levels– Windows – Linux: 32bit, 64bit– Unix– Scale up, scale out

What are your optimization options? Are tools available? Ex. Intel Parallel Studio

Consider Resource Usage when the

Platform is Virtualized!


Platform – Virtualization

On a machine that has much less available Memory and/or CPU, this graph will look different.


Memory Usage Scalability – from the Lab

Application #1

Application #2

Application #3

Application #4

Application #5

Still consumes less Memory!!


Virtualization Update

Updated Performance Lab to scale Virtual machines to measure VM Impact.

VM1 VM2 VM3 Total Comparison

DD Comp DD Comp DD Comp DD Comp VM1 VM2 VM3 Total

1 3559.7 2514.7 3848.1 2575.2 4334 2305 11741.8 7394.9 41.60% 49.40% 88.00% 58.80%

2 5974.4 4029 6002.3 4155.9 7006 3162.7 18982.7 11347.6 48.30% 44.40% 121.50% 67.30%

3 7778.9 4807.6 8276.3 4846.4 8117.6 3631.4 24172.8 13285.4 61.80% 70.80% 123.50% 82.00%

4 9645.5 5329.6 9809.8 5191 8593.5 3699 28048.8 14219.6 81.00% 89.00% 132.30% 97.30%

5 10866.2 5567.9 11031.7 5678.6 9163.5 3832.2 31061.4 15078.7 95.20% 94.30% 139.10% 106.00%

6 11921.4 4426.1 12037.2 4493.3 11259.7 4146.8 35218.3 13066.2 169.30% 167.90% 171.50% 169.50%

7 12798.3 5636.2 12986.4 6312 12401.2 4451.7 38185.9 16399.9 127.10% 105.70% 178.60% 132.80%

8 13718.4 6317.8 13851.1 5820.3 13224.6 3259.7 40794.1 15397.8 117.10% 138.00% 305.70% 164.90%

9 14429.7 6230.1 14601.7 6489 14135.6 3905.4 43167 16624.5 131.60% 125.00% 262.00% 159.70%

10 15228.7 6294.1 15316.7 6277 14648.6 3329.3 45194 15900.4 142.00% 144.00% 340.00% 184.20%

The Competition needs 3 virtual machines to achieve the same performance as 1 of our virtual machines!

+ + =


6277rps

15229rps 14649rps15317rps

Virtualized Optimizations give you options!

Increase throughput Machine Consolidation

6294rps 3329rps

Total = 15900 rps

300% Throughput Increase! Over 3x Cost and space savings!

Total = 45194 rps

-OR-

Add Progress | DataDirect


Application Code & Infrastructure

Faster communication– Caching & Object Reuse

(avoiding comm!)

Increase Parallelism– Threading, Concurrency – Network async streams– Not for basic reporting!

Minimize Slow operations– Disk I/O, blocking operations

Re-architect when ?– Reference architectures

Tools – latest versions

It’s like the Tower of Babel; we have great infrastructure… but we still can’t

communicate


Network

Physical Location

Security is expensive

How many Hops?

At what speed?

Co-locate, Same country, City, etc..


The Network has much to tune…


The Driver is the network


Database

Utilize OpenEdge Tools if available– E.g., promon statement cache options

Data Layout– Tune your SQL queries– Eliminate unnecessary calls and scans– Save MetaData

Memory tuning– Buffer sizes, pre-fetch, etc.

Disk Usage– Striping, storing more data in memory

Logging and health monitors – look in the OpenEdge documentation


OpenEdge sql techniques – better connectivity

Quick ordering– “SELECT TOP”

Minimize round trips– Stored procedures

Capturing preferred query form– Views

Managing variable sized data– dbtool for column width– Network buffer memory impact– Rows per network message

Managing concurrency effects– Lockwait timeout


SELECT TOP – Order and Time

db

App

TOP ordering

Optimized ordering

ORDER BY

Sorting machine


Push network traffic

to server

Java Stored Procedure – logic on the server

AppStored Proc

SQL

db


Pushing best query elements into sql views

Views can be a toolbox of nicely working query components– Pre-fab join relationships for naturally related tables

• The right tables with the right join predicates• The right WHERE clause predicates to choose correct data• Avoid those queries missing a vital WHERE predicate

Views can be a storehouse of totally built queries– End users pick the view vs. end users built query from scratch

Accelerate those ad-hoc reports with views as building blocks

Views can provide security– Expose only “need to know” tables, columns– Screen out data which must be private or secured away

Simple to setup and use– CREATE VIEW myview AS <regular SELECT statement>– SELECT * FROM myview WHERE myview.region = ‘NW’


Failing transactions – the problem

12345,Smith Trucking, Westford,MA,02423,…..

Customer table

UPDATE customerSET city = ‘Northboro’

WHERE custnum = 12345

SELECT *FROM customerWHERE custnum BETWEEN 10000 AND 20000

lockWaitTimeout expires


Failing transactions – some help

Global lockwait timeout– Environment variable PROSQL_LOCKWAIT_TIMEOUT

• export PROSQL_LOCKWAIT_TIMEOUT=30• Significant shortcomings

JDBC Connection property– “lockWaitTimeout”– As of 10.2B04

String url = "jdbc:datadirect:openedge://localhost:9878;" + "databaseName=mys2k;lockWaitTimeout=60";

Connection conn = DriverManager.getConnection(url, ….. );

FUTURE: ODBC Connection property– “lockWaitTimeout”, same format– Target delivery in OE 11.2– In ODBC Data Admin “Advanced” Tab


Power of the ABL– ABL: size unlimited – TEXT– Sql: size limited – VARCHAR (61)

Result: “over-stuffed” columns• OE sql error: “Column XXX has value exceeding it’s max length”

Server side fixes– Data Dictionary

• Options - SQL Properties - Adjust Field Width

– OE SQL as DBA• ALTER TABLE ALTER COL SET PRO_SQL_WIDTH …..

– dbtool• High performance, multi-threaded db maintenance• Menu choice: SQL Width Scan w/Fix Option• Online schema updater

Coping with column sizes


Client side fixes– Find column which has size problem from error text

• “Column Name in table PUB.Customer has value exceeding its max length or precision.”

– Change sql statement column reference to “substring()”• SELECT Name, city, state, country FROM pub.customer ……• SELECT SUBSTR( Name, 1, 84) AS ‘Name’, city, state, country, ……

– OR, truncate over-stuffed value to column defined size• SELECT SUBSTR( Name, 1) AS ‘Name’, city, state, country, ……

Coping with column sizes - more


32K * 100 cols * 50 rows = 160M

Coping with column sizes – maxing out

App

db

Table2 result

set rows

Table1 result set

rows

Table3 result

set rows

Query

network buffers


New Challenges & Research in 2012

More 2012 Enterprise Performance Research– Caching, Concurrency, Cloud deployment

Mobile devices – 3g to 4g to wiMax– Hardware upgrades coming fast

Services still the norm – what’s next?– SaaS– PaaS

SQL vs NoSQL

“Cloud”


In Summary

Humans continue to create copious amounts of data at breakneck speed. This data must be cataloged, sorted, and queried – the latest improvements in data access middleware will ensure this remains scalable for your applications in the enterprise!

R&D trends in 2011 give way to actionable development changes in 2012. Follow the P.A.N.D.Da model to keep your application up to date!


For More Information

Progress|DataDirect Website: www.datadirect.com

More than a Driver: http://web.datadirect.com/products/feature-overview/data-connectivity/anatomy/index.html

Progress|DataDirect Performance Data: http://web.datadirect.com/products/feature-overview/data-connectivity/performance.html

Progress|DataDirect Virtualization: http://web.datadirect.com/solutions/virtualization/virtualization.html

Documentation: http://www.datadirect.com/support/product_info/proddoc_product/index.ssp

Contact Us: http://web.datadirect.com/contact-us/request_information.html

NA Tel Number: +1-800-876-3101

Intl Tel Number: +44 (0) 1753-218 930

http://www.datadirect.com/

http://web.datadirect.com/products/feature-overview/data-connectivity/anatomy/index.html

http://web.datadirect.com/products/feature-overview/data-connectivity/performance.html

http://web.datadirect.com/products/feature-overview/data-connectivity/performance.html

http://web.datadirect.com/solutions/virtualization/virtualization.html

http://www.datadirect.com/support/product_info/proddoc_product/index.ssp

http://www.datadirect.com/support/product_info/proddoc_product/index.ssp

http://web.datadirect.com/contact-us/request_information.html

http://web.datadirect.com/contact-us/request_information.html

Documents

© 2012 Progress Software Corporation. All rights reserved. 1 Click to edit Master title style Remove Performance Bottlenecks in Your SQL Connections -