Upload
rosemary-butt
View
218
Download
0
Embed Size (px)
Citation preview
Finding the Performance Bottlenecks in Your Application
Ian Jones and Roger Schrag
Database Specialists, Inc.
www.dbspecialists.com
IOUG-A Live! 1999
Paper #158
Finding the Bottleneck: Half the Battle in Tuning
• One bad SQL statement can spoil performance
• Too much code to take the “let’s tune every statement” approach
• DBA can’t be familiar with every line of code
Zero in on the Bottleneck
• Use the v$ views
• Use SQL Trace and timed statistics
• Use GUI tools
Today’s Presentation
• Half a dozen real-life examples– scenario– command-line efforts– resolution
• Demo of GUI tools– Enterprise Manager– freeware
The Terrifyingly Slow EDI Load• A transportation company used EDI to exchange data with
customers. They loaded EDI files into a temp table with SQL*Loader and then ran a 1500 line PL/SQL stored procedure to validate the data and update application tables.
• As more customers began sending EDI files, the PL/SQL stored procedure could no longer keep up. Data validation took as long as 24 hours for some EDI files.
• Where do we start tuning?
Start Load Process and Identify the Database SessionSQL> SELECT sid, serial#, status, username, osuser,
2 module, action
3 FROM v$session;
SID SERIAL# STATUS USERNAME OSUSER MODULE ACTION
---- ------- -------- -------- -------- -------- --------
1 1 ACTIVE oracle
2 1 ACTIVE oracle
3 1 ACTIVE oracle
4 1 ACTIVE oracle
5 1 ACTIVE oracle
6 1 ACTIVE oracle
7 54959 ACTIVE BJENKINS bjenkins de
8 4921 INACTIVE RTHOMAS rthomas de
9 2492 INACTIVE EJOHNSON ejohnson de
45 3415 ACTIVE EDI_LOAD edi SQL*Plus validate
View the Statement Being Executed
SQL> SELECT B.sql_text
2 FROM v$session A, v$sqlarea B
3 WHERE A.sid = 45
4 AND B.address = A.sql_address;
SQL_TEXT
-------------------------------------------------------
SELECT ITEM_ID FROM ITEM_TRANSLATIONS WHERE
SOURCE_ID = :b1 AND SUBSTR(SOURCE_SKU_CODE,1,6) = :b2
AND SYSDATE BETWEEN START_DATE_ACTIVE AND NVL
(END_DATE_ACTIVE, SYSDATE)
We Found a Bottleneck!PROCEDURE edi_validate_and_load
(p_cust_id IN NUMBER)
IS
CURSOR c_get_item_id (cp_cust_id IN NUMBER,
cp_sku IN VARCHAR2) IS
SELECT item_id
FROM item_translations
WHERE source_id = cp_cust_id
AND source_sku_code LIKE cp_sku || '%'
AND SYSDATE BETWEEN start_date_active
AND NVL (end_date_active, SYSDATE);
Disk Array Far Too Busy
• One third of a financial institution’s loan processing department started using a new PowerBuilder application. Response time was acceptable, but disk utilization on the server was at 100%.
• What could be done to reduce I/O so that the response time will still be acceptable after the rest of the department starts using the new application?
Identify the SQL Statements Causing the Most Disk Reads
SELECT sql_text, disk_reads, executions,
disk_reads / DECODE (executions, 0, 1, executions)
reads_per_exec
FROM v$sqlarea
ORDER BY reads_per_exec;
SELECT sql_text, disk_reads, executions,
disk_reads / DECODE (executions, 0, 1, executions)
reads_per_exec
FROM v$sqlarea
ORDER BY disk_reads;
SELECT sql_text, buffer_gets, executions,
buffer_gets / DECODE (executions, 0, 1, executions)
gets_per_exec
FROM v$sqlarea
ORDER BY buffer_gets;
Part of the Query Results…SQL_TEXT
------------------------------------------------------------
DISK_READS EXECUTIONS READS_PER_EXEC
---------- ---------- --------------
SELECT P.PRODUCT_DESC, CP.PRODUCT_ID, UPPER (:b1) CLIENT_ID
FROM CLIENT_PRODUCT CP, PRODUCT P WHERE CP.PRODUCT_ID =
P.PRODUCT_ID AND (UPPER (CP.CLIENT_ID),
CP.VALID_CLIENT_LEVEL_ID) IN (SELECT UPPER (:b1),
CA.VALID_CLIENT_LEVEL_ID FROM CLIENTS C, CLIENT_ADDRESS CA
WHERE UPPER (C.CLIENT_ID) = UPPER(:b1) AND C.CLIENT_ID
= CA.CLIENT_ID)
208734602 18657 11188.0046
The Same Query Formatted for Readability
SELECT P.product_desc, CP.product_id,
UPPER (:b1) client_id
FROM client_product CP, product P
WHERE CP.product_id = P.product_id
AND (UPPER (CP.client_id), CP.valid_client_level_id) IN
(SELECT UPPER (:b1), CA.valid_client_level_id
FROM clients C, client_address CA
WHERE UPPER (C.client_id) = UPPER(:b1)
AND C.client_id = CA.client_id)
A case-insensitive query turns out to be very inefficient. Now we know what to tune.
The So-Called “Locking Problem”
• A developer coded a PL/SQL function called compute_irr for computing internal rate of return. Response times varied widely. The developer claimed there was a locking problem on the database.
• What caused the slow performance in compute_irr?
SQL> SELECT sid, serial#, status, username, osuser,
2 module, action
3 FROM v$session;
SID SERIAL# STATUS USERNAME OSUSER MODULE ACTION
---- ------- -------- -------- -------- -------- --------
1 1 ACTIVE oracle
2 1 ACTIVE oracle
3 1 ACTIVE oracle
4 1 ACTIVE oracle
5 1 ACTIVE oracle
6 1 ACTIVE oracle
9 2041 INACTIVE GL swatkins
10 4284 INACTIVE APPLSYS applmgr
62 7219 ACTIVE RSCHRAG rschrag SQL*Plus
63 7394 INACTIVE FA rschrag SQL*Plus
Start SQL*Plus and Identify the Database Session
Check Session Statistics Before Calling compute_irr
SQL> SELECT A.name, B.value
2 FROM v$statname A, v$sesstat B
3 WHERE B.statistic# IN (12, 37, 38, 39, 119, 123,
4 139, 140, 141)
5 AND B.sid = 63
6 AND A.statistic# = B.statistic#;
NAME VALUE
------------------------------ ----------
CPU used by this session 1292
db block gets 10186
consistent gets 86810
physical reads 346
table scans (long tables) 0
table scan rows gotten 1054
sorts (memory) 826
sorts (disk) 0
sorts (rows) 4693
Check Session Statistics Again While compute_irr Is Running
SQL> SELECT A.name, B.value
2 FROM v$statname A, v$sesstat B
3 WHERE B.statistic# IN (12, 37, 38, 39, 119, 123,
4 139, 140, 141)
5 AND B.sid = 63
6 AND A.statistic# = B.statistic#;
NAME VALUE
------------------------------ ----------
CPU used by this session 7274
db block gets 10294
consistent gets 86813
physical reads 346
table scans (long tables) 0
table scan rows gotten 1054
sorts (memory) 826
sorts (disk) 0
sorts (rows) 4693
Session Statistic Before After 60Seconds
After 120Seconds
CPU Used by this session 1292 7274 13255db block gets 10186 10294 10294consistent gets 86810 86813 86813physical reads 346 346 346table scans (long tables) 0 0 0table scan rows gotten 1054 1054 1054sorts (memory) 826 826 826sorts (disk) 0 0 0sorts (rows) 4693 4693 4693
Compute the Deltas
Look for an infinite loop that involves no SQL statements.
Where Is the I/O Coming From?
• An application that performed well in a demo database quickly became I/O-bound when a significant amount of data was loaded into the database. It seemed as if there was a huge amount of I/O even with no users logged on to the application.
• What could be causing so much I/O activity?
Look at Physical Reads Instance-Wide and Per Session
SQL> SELECT name || ' (instance-wide)', value
2 FROM v$sysstat
3 WHERE statistic# = 39
4 UNION ALL
5 SELECT 'sid = ' || TO_CHAR (sid), value
6 FROM v$sesstat
7 WHERE statistic# = 39;
And the Results Are...NAME VALUE
-------------------------------- ----------
physical reads (instance-wide) 6048399
sid = 1 0
sid = 2 0
sid = 3 0
sid = 4 0
sid = 5 0
sid = 6 23120
sid = 7 186
sid = 8 0
sid = 9 18984
sid = 12 1375
sid = 13 5830219
sid = 17 27821
Investigate Suspicious Session 13SQL> SELECT sid, serial#, status, username, osuser,
2 module, action
3 FROM v$session
4 WHERE sid = 13;
SID SERIAL# STATUS USERNAME OSUSER MODULE ACTION
---- ------- -------- --------- -------- -------- --------
13 829 ACTIVE APPSCHEMA daemon EXTRACT GET
SQL> SELECT A.name, B.value
2 FROM v$statname A, v$sesstat B
3 WHERE B.statistic# IN (12, 37, 38, 39, 119, 123,
4 139, 140, 141)
5 AND B.sid = 13
6 AND A.statistic# = B.statistic#;
Suspicious Session 13 (continued)NAME VALUE
------------------------------ ----------
CPU used by this session 0
db block gets 5928722
consistent gets 293
physical reads 5873918
table scans (long tables) 575
table scan rows gotten 72400000
sorts (memory) 3
sorts (disk) 0
sorts (rows) 4210
What SQL Is Session 13 Executing?SQL> SELECT B.sql_text
2 FROM v$session A, v$sqlarea B
3 WHERE B.address = A.sql_address
4 AND A.sid = 13;
SQL_TEXT
------------------------------------------------------
SELECT * FROM TRANSACTIONS WHERE EXTRACTED = 'N’
Let me guess! The transactions table is about 10,000 blocks in size and the extracted column is not indexed.
“Fast” Reports That Were Too Slow
• A software vendor built an application using Oracle, Developer/2000, and a third-party reporting tool. A key report took about six seconds to complete, but consider that users will run the report in batches of several thousand.
• The report has thousands of lines of spaghetti code. How do you figure out what's slowing it down?
Build a Version of the Report That Enables SQL Trace
DECLARE
c INTEGER;
i INTEGER;
BEGIN
c := dbms_sql.open_cursor;
dbms_sql.parse (c,
'ALTER SESSION SET TIMED_STATISTICS=TRUE',
dbms_sql.native);
i := dbms_sql.execute (c);
dbms_sql.close_cursor (c);
dbms_session.set_sql_trace (TRUE);END;
Trace only what you need to trace!
Run the Report, Fetch the Trace File, Run TKPROF On It
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS
call count cpu elapsed disk query
------- ------ -------- ---------- ---------- ----------
Parse 38 3.01 4.22 12 591
Execute 38 0.40 0.65 57 218
Fetch 41 0.57 0.77 91 294
------- ------ -------- ---------- ---------- ----------
total 117 3.98 5.64 160 1103
Misses in library cache during parse: 30
Look into using bind variables to reduce parsing.
The Chameleon Application
• An application ran well on a test database loaded with a full set of production data. But when the application was deployed in production, queries took over a minute to complete. In the test environment these same queries completed in under ten seconds.
• Policies forbid modifying code in production.
• What could be causing the application to run slower in production?
Find a Power User to Reproduce the Slow Behavior
• Identify the database session:
SQL> SELECT sid, serial#, status, username, osuser, 2 module, action
3 FROM v$session
4 WHERE username = 'MARYD';
SID SERIAL# STATUS USERNAME OSUSER MODULE ACTION
---- ------- -------- --------- -------- -------- --------
17 9172 INACTIVE MARYD mbd frontend query
Enable Timed Statistics Temporarily
SQL> ALTER SYSTEM SET TIMED_STATISTICS = TRUE;
System altered.
SQL>
Enable SQl Trace Just Before the Query Is Launched
SQL> BEGIN
2 dbms_system.set_sql_trace_in_session (17, 9172, TRUE);
3 END;
4 /
PL/SQL procedure successfully completed.
SQL>
Disable SQL Trace When the Query Is Finished
SQL> BEGIN
2 dbms_system.set_sql_trace_in_session (17, 9172, FALSE);
3 END;
4 /
PL/SQL procedure successfully completed.
SQL>
SQL> ALTER SYSTEM SET TIMED_STATISTICS = FALSE;
System altered.
SQL>
Fetch the Trace File and Run TKPROFcall count cpu elapsed disk query current rows
------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 1.44 1.45 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 17 68.39 68.54 0 1878 2 254
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 19 69.83 69.99 0 1878 2 254
Misses in library cache during parse: 1
Optimizer goal: RULE
Parsing user id: 142 (BUILD4P2)
Rows Execution Plan
------- ---------------------------------------------------
0 SELECT STATEMENT GOAL: RULE
0 MERGE JOIN (OUTER)
254 SORT (JOIN)
115 NESTED LOOPS (OUTER)
253 NESTED LOOPS (OUTER)
254 TABLE ACCESS GOAL: ANALYZED (BY ROWID) OF 'MNME'
114539 INDEX GOAL: ANALYZED (FULL SCAN) OF 'MNME_I1'
(UNIQUE)
253 TABLE ACCESS GOAL: ANALYZED (BY ROWID) OF 'LCONTYPE'
254 INDEX GOAL: ANALYZED (UNIQUE SCAN) OF 'LCONTYPE_PK'
(UNIQUE)
115 TABLE ACCESS GOAL: ANALYZED (BY ROWID) OF 'MTAX'
254 INDEX GOAL: ANALYZED (UNIQUE SCAN) OF 'MTAX_I1'
(UNIQUE)
129 SORT (JOIN)
129 TABLE ACCESS GOAL: ANALYZED (FULL) OF 'LPOST’
The optimizer can behave differently from one Oracle version to the next.
GUI Tools Demo
To Find the Bottlenecks in Your Applications:
• Monitor v$sqlarea
• Monitor v$sysstat and v$sesstat
• Use SQL Trace judiciously
• Consider using GUI tools
ResourcesOracle Server Tuning
- Overview of the tuning process- How to use SQL Trace and TKPROF
Oracle Server Reference- Descriptions of all v$ views
High Performance SQL Tuning by Guy Harrison- Lots of tuning tips- Discussion of GUI tools available on the Internet
www.dbspecialists.com/present.html- Download this presentation- Download a companion white paper
Contact Information
Ian Jones: [email protected]
Roger Schrag: [email protected]
Database Specialists, Inc.
388 Market Street, Suite 400
San Francisco, CA 94111
415/344-0500