106/05/08
Statspack
Kyle Hailey
http://perfvision.com/ftp/apac2010http://oraclemonitor.com
Copyright @ 2006 Kyle Hailey
Brief HistoryBrief History1989 6 Weakness of Ratios
Ratio based guess work – no waits Utlbstat/Utlestat Lists of rules – no global picture/perspective
1993 7 Power of Waits Time gains and losses clear Undocumented “Expert Only”
1999 8.1.6 Statspack Top 5 Wait Events Filtered out Idle Events Compulsive Tuning Disorder
2001 9iR2 CPU Top 5 Timed Events (includes CPU) Response Time = Service Time (CPU) + Wait Time
2003 10g Simplicity of AAS Average Active Session (AAS) = DB Time / Elapsed Time
2007 11g SQL Row Source
Counters
Timings
Sampling
V6 Problem V6 Problem
How do you shorten your daily commute?Analyze Statistics ?
RPMNumber of rotations of wheels?Average amount of gas in tank?
No !
Solution:Analyze Time
Where do you get blocked?Are there alternatives ?
Oracle 6 didn’t have wait times, but you do !
Ratios, Example v6Ratios, Example v6
Wait timings unavailable for IO Db file sequential read
Db file scattered read
Direct path read
Ratio : Cache Buffer Hit proxy for IO wait time
Unreliable High ratio
Bad – untuned SQL doing mega LIOs
Low ratio
Good - I/Os are coming from UNIX file cache or disk cache
Reliability: IO Wait Times
06/05/08
Wait Events replaced Ratios !
Oracle 7Oracle 7
Waits introduced in version 7Quantum Step
1992 slipped in for a benchmark by kernel architect (now Senior VP) Juan Loaiza in 1992
Benchmark couldn’t find where time was being spent
Wait instrumentation solved the mystery
Originally UndocumentedStill badly documented course should help change that
Experts only
Thus, ratio based tuning dominated
06/05/08
Copyright 2006 Kyle Hailey
Oracle InstrumentationOracle Instrumentation
*$%@!!*$%@!!
Database: Guilty until proven innocent
RedoRedo Lib Lib CacheCache
Buffer Buffer CacheCache
IOIO
LocksLocksNetworkNetwork
CPUCPU
Database
StatspackStatspack
Introduced in 8.1.6by Connie Dialeris
Snapshots Selects from 55 tables and views
x$ structures
v$ views
DBA views
Number of tables has expanded with each version
30 plus tables in Oracle8i
40 plus tables in Oracle9i
55 plus tables in Oracle10g
Calculates Deltas (and ratios)
Reports 1000-2000 lines output, about 30 pages
Copyright @ 2006 Kyle Hailey
Copyright 2006 Kyle Hailey
Statspack: Cheat SheetStatspack: Cheat Sheet
InstallConnect as SYSDBA
@?/rdbms/admin/spcreate.sql
RunExec statspack.snap;
Generate Reports@?/rdbms/admin/spreport.sql
(AWR : @?/rdbms/admins/awrrpt.sql )
Statspack Sections (10g)Statspack Sections (10g)Instance descriptionHost hardwareSnapshot headlineCache informationLoad profileInstance efficiencyShared pool statsTop timed eventsHost CPU loadHost / Instance CPU warningVM activityMemory usageTime model statsRAC statisticsAll wait eventsBackground wait eventsEvent histogramSQL x 9Instance activityLog switchesOS stats
Latch parent and childMutexSegment stats x 8Dictionary cacheLibrary cacheRAC (GES)RAC (CR and CUR
served)RAC (cache xfer x 2)RAC (Remastering)Streams x 7Shared pool advisorJava pool advisorSGA resizingSGA target advisorSGA summarySGA detailSQL memory summaryResource limitsParameters
Session details x 3Tablespace I/OFile I/OFile I/O histogramBuffer poolInstance RecoveryBuffer pool advisoryBuffer busy waitsVarious PGA
summariesPGA histogramPGA advisoryPGA allocation
summaryPGA allocation top
NEnqueue (lock)
activityUndo stats x 2Latch activityLatch miss details
Statspack MethodStatspack Method
06/05/08
Load profile good for having a feel for the application and comparing two periods for changes
Efficiency ratios misleading carry over from version 6 days
Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call Time---------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72CPU time 32 10.16free buffer waits 1,588 15 4.63write complete waits 10 8 2.51log buffer space 306 5 1.51 ----------------------------------------
Summary
Waits
Who/When
Big Picture
Oracle 8Oracle 8
06/05/08
Top 5 Wait Events ~~~~~~~~~~~~~~~~~ Wait % TotalEvent Waits Time (cs) Wt Time-------------------------------------------- ------------ ------------ -------direct path read 4,232 10,827 52.01db file scattered read 6,105 6,264 30.09direct path write 1,992 3,268 15.70control file parallel write 893 198 .95db file parallel write 40 131 .63
Statistic Total per Second per Trans --------------------------------- ---------------- ------------ ------------ CPU used by this session 358,806 130.5 12,372.6
Missing the CPUCPU time = 95%Eliminating all IO gains 3 % !
IO Looks horrible, but wait …
Total time in secs = 108+62+32+1+1+3588=3792
3588/3792=95%
108/3792=3%
Oracle 9Oracle 9
06/05/08
Statspack adds CPU
Method% wait time = potential performance gain
Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time-------------------------------------------- ------------ ----------- --------CPU time 8,026 93.78direct path write 1,930 421 4.92db file scattered read 4,746 69 .80direct path read 77 18 .21db file parallel write 23 11 .13 ------------------------------------------------------------
Max gain is 6.22% by tuning all waits
In other words go tune the top SQL
% gain = Savings/Service TimeService Time = CPU time + Wait Time
Service Time =DB Time in 10g
Problems in 9Problems in 9
1. Lack of Data to solve issues
Examples
1.Buffer busy wait
2. latch free
3.db file scattered read (IO)
4.Enqueue (lock)
Buffer Busy WaitBuffer Busy Wait
06/05/08
Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call Time-------------------------------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72CPU time 32 10.16free buffer waits 1,588 15 4.63write complete waits 10 8 2.51log buffer space 306 5 1.51 ------------------------------------------------------------
Buffer Busy Wait – trying to modify a block a buffer. Who blocks, what kind of buffer, what is the SQL?Buffer wait Statistics DB/Inst: DB4/db4 Snaps: 2-3Class Waits Wait Time (s) Avg Time (ms)------------------ ----- ------------- -------------file header block 264 203 769data block 6,070 162 27undo header 355 0 1
Statspack fails for analysis
Latch FreeLatch Free
06/05/08
Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time-------------------------------------------- ------------ ----------- --------latch free 9,652 760 66.10CPU time 248 21.62PL/SQL lock timer 41 123 10.72SQL*Net message from dblink 681 14 1.22log file parallel write 128 1 .13 -------------------------------------------------------------
What Latch? There are 100sLatch Sleep breakdown for DB: CDB Instance: cdb Snaps: 3 -4-> ordered by misses descLatch Name Requests Misses SleepsSleeps Sleeps 1->4-------------------------- -------------- ----------- ----------- ------------cache buffers chains 8,448,787 649,484 cache buffers chains 8,448,787 649,484 6,9306,930 0/0/0/0/00/0/0/0/0library cache pin 8,405,896 82,915 1,427 81537/1330/4 7/1/0library cache 8,435,488 55,645 1,294 54375/1247/2 2/1/0shared pool 58,626 7 1 6/1/0/0/0
IOIO
06/05/08
Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time------------------------ ------------ ----------- --------db file scattered read 7,549 5 52.26CPU time 2 23.29db file sequential read 884 2 19.11control file sequential read 154 0 2.68control file parallel write 255 0 1.87
Are the disks slow? What files?Is the buffer cache too small?IO – what SQL? Is it because of rollback?
5/7549=0.6msVery fastProblem is SQL
IOIO
06/05/08
Is the buffer cache too small?IO – what SQL?
File IO Stats for DB: ORA9 Instance: ora9 Snaps: 21 -22Tablespace Filename------------------------ ---------------------------------------------------- Av Av Av Av Av Buffer Av Buf Reads Reads/s Rd(ms) Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)-------------- ------- ------ ------- ------------ -------- ---------- ------EXAMPLE D:\ORACLE\ORADATA\ORA9\EXAMPLE01.DBF 49 0 3.13.1 3.0 0 0 0
SYSTEM D:\ORACLE\ORADATA\ORA9\SYSTEM01.DBF 8,331 11 0.7 0.7 7.3 3 0 0
UNDOTBS1 D:\ORACLE\ORADATA\ORA9\UNDOTBS01.DBF 5 0 8.08.0 1.0 0 0 0
Are the disks slow? What files?
< 20ms
IOIO
06/05/08
IO – what SQL? Is it because of rollback?
Buffer Pool Advisory for DB: ORA9 Instance: ora9 End Snap: 22 Size for Size Buffers for Est Physical EstimatedP Estimate (M) Factr Estimate Read Factor Physical Reads--- ------------ ----- ---------------- ------------- ------------------D 4 .1 979 1.05 67,973D 16 .5 3,916 1.00 64,668D 32 1.0 7,832 1.00 64,612D 32 1.0 7,832 1.00 64,612D 48 1.5 11,748 1.00 64,612D 64 2.0 15,664 1.00 64,612D 68 2.1 16,643 1.00 64,612D 72 2.3 17,622 0.30 19,132D 72 2.3 17,622 0.30 19,132D 76 2.4 18,601 0.30 19,132D 80 2.5 19,580 0.30 19,132
Is the buffer cache too small?
IOIO
06/05/08
Sorted by Reads, not Time
(reads proxy for IO wait time)
SQL ordered by Reads for DB: ORA9 Instance: ora9 Snaps: 21 -22 CPU Elapsd Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value--------------- ------------ -------------- ------ -------- --------- ---------- 60,197 4 15,049.3 98.4 0.48 5.04 1432830824Module: SQL*Plusselect count(*) from sys.source$
666 1 666.0 1.1 0.70 1.46 2893531678Module: SQL*Plusselect count(*) from all_objects
232 24 9.7 0.4 0.53 0.97 238087931select t.schema, t.name, t.flags, q.name from system.aq$_queue_tables t, sys.aq$_queue_table_affinities aft, system.aq$_que
What SQL?
Row Locks 10g+Row Locks 10g+
06/05/08
op 5 Timed Events Avg %Total~~~~~~~~~~~~~~~~~~ wait CallEvent Waits Time (s) (ms) Time----------------------------------------- ------------ ----------- ------ ------enq: TX - row lock contention 59 160 2714 41.8PL/SQL lock timer 4 117 29291 30.6CPU time 28 7.2buffer busy waits 1,217 18 15 4.7log file parallel write 422 11 27 3.0
Who is waitingWho is blockingWhat is the SQLWhat is the row?
2104/21/23
Current MethodCurrent Method
1000 lines of data / 30 pages
What do you look at ?
Top 5 Timed Events?
Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time------------------------ ------------ ----------- --------CPU time 11,330 81.95log file sync 163,725 1,012 7.32enqueue 4,486 615 4.45latch free 28,786 236 1.71db file sequential read 1,420,355 149 1.08 -----------------------------------------
Jonathan Lewis - http://www.jlcomp.demon.co.uk/statspack_01.html
188 minutes CPUElapsed 15 minutes48 Processors or74% idle
StatspackStatspack
Top 5 Timed Events missingCPU Count
Elapsed Time
As well as drill down diagnostic data
Take a step back, take deep breath, and start over
Start with Typical Problems Identify Solutions Design a Visual Representation
Put intelligence into the interface
06/05/08
OEM 10g OEM 10g
Simple, Powerful
DatabaseLoad
Load Source:SQLSessionWait