106/05/08

Statspack

Kyle Hailey

http://oraclemonitor.com

Copyright @ 2006 Kyle Hailey

Brief HistoryBrief History1989 6 Weakness of Ratios

Ratio based guess work – no waits Utlbstat/Utlestat Lists of rules – no global picture/perspective

1993 7 Power of Waits Time gains and losses clear Undocumented “Expert Only”

1999 8.1.6 Statspack Top 5 Wait Events Filtered out Idle Events Compulsive Tuning Disorder

2001 9iR2 CPU Top 5 Timed Events (includes CPU) Response Time = Service Time (CPU) + Wait Time

2003 10g Simplicity of AAS Average Active Session (AAS) = DB Time / Elapsed Time

2007 11g SQL Row Source

Counters

Timings

Sampling

V6 Problem V6 Problem

How do you shorten your daily commute?Analyze Statistics ?

RPMNumber of rotations of wheels?Average amount of gas in tank?

No !

Solution:Analyze Time

Where do you get blocked?Are there alternatives ?

Oracle 6 didn’t have wait times, but you do !

Ratios, Example v6Ratios, Example v6

Wait timings unavailable for IO Db file sequential read

Db file scattered read

Direct path read

Ratio : Cache Buffer Hit proxy for IO wait time

Unreliable High ratio

Bad – untuned SQL doing mega LIOs

Low ratio

Good - I/Os are coming from UNIX file cache or disk cache

Reliability: IO Wait Times

06/05/08

Wait Events replaced Ratios !

Oracle 7Oracle 7

Waits introduced in version 7Quantum Step

1992 slipped in for a benchmark by kernel architect (now Senior VP) Juan Loaiza in 1992

Benchmark couldn’t find where time was being spent

Wait instrumentation solved the mystery

Originally UndocumentedStill badly documented course should help change that

Experts only

Thus, ratio based tuning dominated

06/05/08

Copyright 2006 Kyle Hailey

Oracle InstrumentationOracle Instrumentation

*$%@!!*$%@!!

Database: Guilty until proven innocent

RedoRedo Lib Lib CacheCache

Buffer Buffer CacheCache

IOIO

LocksLocksNetworkNetwork

CPUCPU

Database

StatspackStatspack

Introduced in 8.1.6by Connie Dialeris

Snapshots Selects from 55 tables and views

x$ structures

v$ views

DBA views

Number of tables has expanded with each version

30 plus tables in Oracle8i

40 plus tables in Oracle9i

55 plus tables in Oracle10g

Calculates Deltas (and ratios)

Reports 1000-2000 lines output, about 30 pages

Copyright @ 2006 Kyle Hailey

Copyright 2006 Kyle Hailey

Statspack: Cheat SheetStatspack: Cheat Sheet

InstallConnect as SYSDBA

@?/rdbms/admin/spcreate.sql

RunExec statspack.snap;

Generate Reports@?/rdbms/admin/spreport.sql

(AWR : @?/rdbms/admin/awrrpt.sql )

Statspack Sections (10g)Statspack Sections (10g)Instance descriptionHost hardwareSnapshot headlineCache informationLoad profileInstance efficiencyShared pool statsTop timed eventsHost CPU loadHost / Instance CPU warningVM activityMemory usageTime model statsRAC statisticsAll wait eventsBackground wait eventsEvent histogramSQL x 9Instance activityLog switchesOS stats

Latch parent and childMutexSegment stats x 8Dictionary cacheLibrary cacheRAC (GES)RAC (CR and CUR

served)RAC (cache xfer x 2)RAC (Remastering)Streams x 7Shared pool advisorJava pool advisorSGA resizingSGA target advisorSGA summarySGA detailSQL memory summaryResource limitsParameters

Session details x 3Tablespace I/OFile I/OFile I/O histogramBuffer poolInstance RecoveryBuffer pool advisoryBuffer busy waitsVarious PGA

summariesPGA histogramPGA advisoryPGA allocation

summaryPGA allocation top

NEnqueue (lock)

activityUndo stats x 2Latch activityLatch miss details

Statspack MethodStatspack Method

06/05/08

Load profile good for having a feel for the application and comparing two periods for changes

Efficiency ratios misleading carry over from version 6 days

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call Time---------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72CPU time 32 10.16free buffer waits 1,588 15 4.63write complete waits 10 8 2.51log buffer space 306 5 1.51 ----------------------------------------

Summary

Waits

Who/When

Big Picture

Oracle 8Oracle 8

06/05/08

Top 5 Wait Events ~~~~~~~~~~~~~~~~~ Wait % TotalEvent Waits Time (cs) Wt Time-------------------------------------------- ------------ ------------ -------direct path read 4,232 10,827 52.01db file scattered read 6,105 6,264 30.09direct path write 1,992 3,268 15.70control file parallel write 893 198 .95db file parallel write 40 131 .63

Statistic Total per Second per Trans --------------------------------- ---------------- ------------ ------------ CPU used by this session 358,806 130.5 12,372.6

Missing the CPUCPU time = 95%Eliminating all IO gains 3 % !

IO Looks horrible, but wait …

Total time in secs = 108+62+32+1+1+3588=3792

3588/3792=95%

108/3792=3%

Oracle 9Oracle 9

06/05/08

Statspack adds CPU

Method% wait time = potential performance gain

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time-------------------------------------------- ------------ ----------- --------CPU time 8,026 93.78direct path write 1,930 421 4.92db file scattered read 4,746 69 .80direct path read 77 18 .21db file parallel write 23 11 .13 ------------------------------------------------------------

Max gain is 6.22% by tuning all waits

In other words go tune the top SQL

% gain = Savings/Service TimeService Time = CPU time + Wait Time

Service Time =DB Time in 10g

Problems in 9Problems in 9

1. Lack of Data to solve issues

Examples

1.Buffer busy wait

2. latch free

3.db file scattered read (IO)

4.Enqueue (lock)

Buffer Busy WaitBuffer Busy Wait

06/05/08

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Call Time-------------------------------------------- ------------ ----------- ---------buffer busy waits 2,748 250 78.72CPU time 32 10.16free buffer waits 1,588 15 4.63write complete waits 10 8 2.51log buffer space 306 5 1.51 ------------------------------------------------------------

Buffer Busy Wait – trying to modify a block a buffer. Who blocks, what kind of buffer, what is the SQL?Buffer wait Statistics DB/Inst: DB4/db4 Snaps: 2-3Class Waits Wait Time (s) Avg Time (ms)------------------ ----- ------------- -------------file header block 264 203 769data block 6,070 162 27undo header 355 0 1

Statspack fails for analysis

Latch FreeLatch Free

06/05/08

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time-------------------------------------------- ------------ ----------- --------latch free 9,652 760 66.10CPU time 248 21.62PL/SQL lock timer 41 123 10.72SQL*Net message from dblink 681 14 1.22log file parallel write 128 1 .13 -------------------------------------------------------------

What Latch? There are 100sLatch Sleep breakdown for DB: CDB Instance: cdb Snaps: 3 -4-> ordered by misses descLatch Name Requests Misses SleepsSleeps Sleeps 1->4-------------------------- -------------- ----------- ----------- ------------cache buffers chains 8,448,787 649,484 cache buffers chains 8,448,787 649,484 6,9306,930 0/0/0/0/00/0/0/0/0library cache pin 8,405,896 82,915 1,427 81537/1330/4 7/1/0library cache 8,435,488 55,645 1,294 54375/1247/2 2/1/0shared pool 58,626 7 1 6/1/0/0/0

IOIO

06/05/08

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time------------------------ ------------ ----------- --------db file scattered read 7,549 5 52.26CPU time 2 23.29db file sequential read 884 2 19.11control file sequential read 154 0 2.68control file parallel write 255 0 1.87

Are the disks slow? What files?Is the buffer cache too small?IO – what SQL?

5/7549=0.6msVery fastProblem is SQL

IOIO

06/05/08

Is the buffer cache too small?IO – what SQL?

File IO Stats for DB: ORA9 Instance: ora9 Snaps: 21 -22Tablespace Filename------------------------ ---------------------------------------------------- Av Av Av Av Av Buffer Av Buf Reads Reads/s Rd(ms) Rd(ms) Blks/Rd Writes Writes/s Waits Wt(ms)-------------- ------- ------ ------- ------------ -------- ---------- ------EXAMPLE D:\ORACLE\ORADATA\ORA9\EXAMPLE01.DBF 49 0 3.13.1 3.0 0 0 0

SYSTEM D:\ORACLE\ORADATA\ORA9\SYSTEM01.DBF 8,331 11 0.7 0.7 7.3 3 0 0

UNDOTBS1 D:\ORACLE\ORADATA\ORA9\UNDOTBS01.DBF 5 0 8.08.0 1.0 0 0 0

Are the disks slow? What files?

< 10ms

IOIO

06/05/08

IO – what SQL?

Buffer Pool Advisory for DB: ORA9 Instance: ora9 End Snap: 22 Size for Size Buffers for Est Physical EstimatedP Estimate (M) Factr Estimate Read Factor Physical Reads--- ------------ ----- ---------------- ------------- ------------------D 4 .1 979 1.05 67,973D 16 .5 3,916 1.00 64,668D 32 1.0 7,832 1.00 64,612D 32 1.0 7,832 1.00 64,612D 48 1.5 11,748 1.00 64,612D 64 2.0 15,664 1.00 64,612D 68 2.1 16,643 1.00 64,612D 72 2.3 17,622 0.30 19,132D 72 2.3 17,622 0.30 19,132D 76 2.4 18,601 0.30 19,132D 80 2.5 19,580 0.30 19,132

Is the buffer cache too small?

IOIO

06/05/08

Sorted by Reads, not Time

(reads proxy for IO wait time)

SQL ordered by Reads for DB: ORA9 Instance: ora9 Snaps: 21 -22 CPU Elapsd Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value--------------- ------------ -------------- ------ -------- --------- ---------- 60,197 4 15,049.3 98.4 0.48 5.04 1432830824Module: SQL*Plusselect count(*) from sys.source$

666 1 666.0 1.1 0.70 1.46 2893531678Module: SQL*Plusselect count(*) from all_objects

232 24 9.7 0.4 0.53 0.97 238087931select t.schema, t.name, t.flags, q.name from system.aq$_queue_tables t, sys.aq$_queue_table_affinities aft, system.aq$_que

What SQL?

Row Locks 10g+Row Locks 10g+

06/05/08

op 5 Timed Events Avg %Total~~~~~~~~~~~~~~~~~~ wait CallEvent Waits Time (s) (ms) Time----------------------------------------- ------------ ----------- ------ ------enq: TX - row lock contention 59 160 2714 41.8PL/SQL lock timer 4 117 29291 30.6CPU time 28 7.2buffer busy waits 1,217 18 15 4.7log file parallel write 422 11 27 3.0

Who is waitingWho is blockingWhat is the SQLWhat is the row?

2104/20/23

Current MethodCurrent Method

1000 lines of data / 30 pages

What do you look at ?

Top 5 Timed Events?

Top 5 Timed Events~~~~~~~~~~~~~~~~~~ % TotalEvent Waits Time (s) Ela Time------------------------ ------------ ----------- --------CPU time 11,330 81.95log file sync 163,725 1,012 7.32enqueue 4,486 615 4.45latch free 28,786 236 1.71db file sequential read 1,420,355 149 1.08 -----------------------------------------

Jonathan Lewis - http://www.jlcomp.demon.co.uk/statspack_01.html

188 minutes CPUElapsed 15 minutes48 Processors or74% idle

StatspackStatspack

Top 5 Timed Events missingCPU Count

Elapsed Time

As well as drill down diagnostic data

Take a step back, take deep breath, and start over

Start with Typical Problems Identify Solutions Design a Visual Representation

Put intelligence into the interface

06/05/08

OEM 10g OEM 10g

Simple, Powerful

DatabaseLoad

Load Source:SQLSessionWait