05 WAITS Intro

8/8/2019 05 WAITS Intro

1/41

Part II : Waits Eventsand the

Geeks who love them

Kyle Hailey

http://perfvision.com


2/41

Copyright 2006 Kyle Hailey

Wait Events

Wait EventsWait Events


3/41


4/41


In this Section:

Introduction to WaitsTuning Methodology

Plan of Action

Statspacks, AWR or OEM forCollectionData

Based on Waits

Using Waits to Solve Bottlenecks


5/41


Database is Hung!

Everybody blames the database

Yet 9 out of 10 dbas agree its not the

databaseHow do you prove it to management?

On the off chance its the database, what dowe do?


6/41


Database:

Guilty until proven innocent

*$%@!!*$%@!!


7/41


Oracle Instrumentation

RedoRedo LibLibCacheCache

BufferBufferCacheCache

IOIO

LocksLocks

NetworkNetwork

CPUCPU


8/41


Waits

I/O

Library Cache

Locks

Redo

Buffer Cache

SQL*Net

Wait Areas

Well discuss Waits in these logical database areasWell discuss Waits in these logical database areas


9/41

Copyright 2006 Kyle HaileyCopyright 2006 Kyle Hailey

Wait Tree

WaitsWaits

IOIO

Buffer CacheBuffer Cache

Library CacheLibrary Cache

LockLock

RedoRedo

SQL NetSQL Net

Buffer BusyBuffer Busy

RollbackRollback

Free listsFree lists

IO ReadIO ReadCache LatchesCache Latches

Library CacheLibrary Cache

Shared PoolShared Pool

TX Row LockTX Row Lock

TX ITL LockTX ITL Lock

HW LockHW Lock

Write IOWrite IO

Read IORead IO

Log BufferLog Buffer

Log File SyncLog File Sync

Log FileLog File


10/41


Waits

Introduced in v7Revolutionized tuning

Changed from Ratio Guesswork to empiricalmeasure of time lost to bottlenecks

10g added the crucial addition ASHNot only identifies bottlenecks but

Who (session, service, package, procedure)

Where (CPU, Wait)

When (time)

What (SQL statement)


11/41


Tuning Methodology

1. Machine Run queue (CPU) Check other applications

reduce CPU usage or add CPUs

Paging

Reduce memory usage or add memory

2. Oracle Waits + CPU > Available CPU

Tune waits

CPU 100% Tune SQL

Else low waits, available CPU then Its the application

We are going toWe are going toconcentrate hereconcentrate hereon WAITSon WAITS


12/41


Dependable Tuning Strategy

Determine AAS :1. Run Statspack or AWR Report

Top 5 Timed Events ~50 lines down from top

Need AvailableC

PU Elapsed Time

CPU_COUNT

2. ASH Report : ashrpt.sql

3. OEM 10gPerformance Page does everything

If there is a wait bottleneck tune the wait


13/41


Tuning Methodology Graphics

Relax, its theRelax, its theapplicationapplication

Get to Work!Get to Work!


14/41


Waits beyond OEM

OEM identifies Wait problems

Provides solutions with ADDM sometimes

But What do you do when ADDM isnt sufficient?

What do you do if you dont have OEM 10g?Waits

Need to know about waits

How they work

How to analyze them


15/41


v$active_session_history

When ADDM fails or we dont have ADDM we cancollect the necessary information from

v$active_session_history Session (user, service, client, package, procedure, etc)

SQL statement For IO waits, buffer busy waits and enqueue TX waits

CURRENT_OBJ# ,CURRENT_FILE# ,CURRENT_BLOCK#

Blocking_Session

P1

P2 P3


16/41


What are P1,P2,P3 ?

Each Wait has a 3 parameters P1,P2,P3Give detailed information

Meaning different for each wait

Meaning definitions in V$event_name

col parameter1 for a10col parameter1 for a10

col parameter2 for a10col parameter2 for a10col parameter3 for a10col parameter3 for a10

select parameter1 ,parameter2 , parameter3select parameter1 ,parameter2 , parameter3from v$event_namefrom v$event_namewhere name = '&1';where name = '&1';


17/41


Wait Arguments Example

NAME PARAMETER1 PARAMETER2 PARAMETER

------------------------------ ----------- --------------- ---------------

latch: cache buffers chains address number tries

free buffer waits file# block# set-id# buffer busy waits file# block# class#

latch: redo copy address number tries

log buffer space

switch logfile command

log file sync buffer#

db file sequential read file# block# blocks

enq: TM - contention name|mode object # table/partition

undo segment extension segment#

enq: TX - row lock contention name|mode usn


18/41


Wait Analysis requires p1,p2,p3

Of the top 30 wait events 8 can be solvedwithout ASH

The rest needSQL

P1,P2,P3

Statspack , AWR fail

free buffer waitslog buffer spacelog file switch (archiving needed)

log file switch (checkpoint incomplete)log file switch completionlog file syncswitch logfile commandwrite complete waits

Example hard waitsExample hard waitsBuffer busy waitBuffer busy waitRow cache lockRow cache lockLatch freeLatch freerow lock contentionrow lock contentionLatch: cache buffers chainsLatch: cache buffers chains


19/41

Wait Analysis

SQL Most often the tuning answer lies in looking at what the application is

doing, and changing it

Parameters Find extended wait information

Parameter1, Parameter2, Parameter3

Defined in v$event_name

Guess Work Sometimes the wait events that are found are not in the

documentation and it takes some educated guesswork to figure outthe problem


20/41


Difficult Waits

These 4 waits have multiple causes

Latches p2 = latch # (p1= address, p3= tries)

Locks p1 = lock type and mode ( p2 = id1, p3= id2)

Buffer Busy p3 = block class#, p1= file, p2=block

(in 9i p3 was the bbw type)

Row Cache Lock p1 = cache id (p2 = mode, p3=request)


21/41


Example ASH Query

Select ash.p1,Select ash.p1,

ash.p2,ash.p2,CURRENT_OBJ#||' '||CURRENT_OBJ#||' '||o.object_nameo.object_name objnobjn,,o.object_typeo.object_type otypeotype,,CURRENT_FILE#CURRENT_FILE# filenfilen,,CURRENT_BLOCK#CURRENT_BLOCK# blocknblockn,,ash.SQL_IDash.SQL_ID,,w.classw.class ||' '||||' '||to_charto_char(ash.p3)(ash.p3) block_typeblock_type

fromfrom v$active_sv$active_session_historyession_history ash,ash,( select( select rownumrownum class#, class fromclass#, class from v$waitstatv$waitstat ) w,) w,all_objectsall_objects oo

wherewhere event='buffer busy waits'event='buffer busy waits'andand w.classw.class#(+)=ash.p3#(+)=ash.p3andand o.object_ido.object_id (+)=(+)= ash.CURRENT_OBJash.CURRENT_OBJ##andand ash.session_stateash.session_state='WAITING'='WAITING'andand ash.sample_timeash.sample_time >> sysdatesysdate -- &1/(60*24)&1/(60*24)

Order byOrder by sample_timesample_timeP1 P2 OBJN OTYPE FILEN BLOCKN SQL_ID BLOCK_TYPEP1 P2 OBJN OTYPE FILEN BLOCKN SQL_ID BLOCK_TYPE---- ------------ ------------------------------------------ ---------- ---------- ------------ -------------------------- --------------

1 112796 66053 BBW_INDEX_VAL_I INDEX 1 112796 6avm49ys4k7t6 data block 11 112796 66053 BBW_INDEX_VAL_I INDEX 1 112796 6avm49ys4k7t6 data block 1

1 112401 66053 BBW_INDEX_VAL_I INDEX 1 112401 5wqps1quuxqr4 data block 11 112401 66053 BBW_INDEX_VAL_I INDEX 1 112401 5wqps1quuxqr4 data block 1




22/41


Waits we will Ignore

One thing that makes waits difficult is knowing whichones to look at and which ones to ignore.

Background

Idle

Resource Manager

Parallel Query

RAC

Good stuff, but not covered in this seminar


23/41


Background & Foreground

Background ProcessesDBWR

LGWR

PMON

SMON

Etc

Foreground ProcessesSQL*Plus

Pro*C

SQL*FormsOracle applications

Only interested in Foreground waits


24/41


Background Waits

ASH

Avoid Background waits in ASH with

V$session_wait joined to v$session

Select from v$active_session_historywhere SESSION_TYPE='FOREGROUND'

select

from v$session s,

v$session_wait wwhere w.sid=s.sid

and s.type='USER'


25/41


Idle Waits

Filtered Out of ASH by default 10g

where wait_class != Idle

Create a list

9i

Create a list with

Documentation List created from 10g

Stats$idle_events from statspack

Select name from v$event_name wherewait_class=Idle;

SQL*Net message from clientSQL*Net message from client


26/41


PQO and Resource Manager

Resource manager throttles user

Creates wait

Obfuscates problems

Parallel Query Wait events are unusable

Save waits are both idle and waits

Parallel Query Waits start with PX or KX PX Deq: Par Recov Reply

PX Deq: Parse Reply

select name from v$event_name wherewait_class='Scheduler';


27/41


RAC Waits

RAC

waits are certainly interesting but will be coveredoutside of this presentation.

You are on your own Check documentation

If you are not using RAC then no worries 10g

9i RAC and OPS waits usually contain the word global

Select event from v$event_name wherewait_class=Cluster;


28/41


Latches

Protect areas of memory from concurrent use Light weight locks

Bit in memory

Atomic processor call

Fast and cheapGone if memory is lost

Often used in cache coherency managementChanges to a data block

Exclusive GenerallySharing reading has been introduced for some latches


29/41


Finding Latches

latch free Covers many latches, find the problem latch by

1. select name from v$latchname where latch# = p1;OR

2. Find highest sleeps in Statspack latch section In 10g, important latches have a wait event

latch: cache buffers chains

latch: shared pool

latch: library cache Latches avoid:Latches avoid: deallocatingdeallocating while someone is accessingwhile someone is accessing reading while someone is modifyingreading while someone is modifying modifingmodifing while someone is modifyingwhile someone is modifying modifingmodifing while someone is readingwhile someone is reading


30/41

Latches and Spin Count

Latch contention is a symptom of a problem

Spin Count

Band aid

Doesnt solve the problem

If there is free CPU Can increase spin count

Underscore parameter in 10g (not recommended to change)

Fix the problem

Instead of using spin countFind the problem and fix it



31/41


Enqueues aka Locks

Enqueue wait covers all locks pre 10

Protect data against concurrent changes

Lock info written into data structures

Block headers

Data blocksWritten in cache structures

Shareable in compatible modes


32/41


Locks 10g

10g breaks all Enqueues out enq: HW - contention Configuration

enq: TM - contention Application

enq: TX - allocate ITL entry Configuration

enq: TX - index contention Concurrency

enq: TX - row lock contention Application enq: UL - contention Application


33/41


Row Cache Lock

Need p1 to see the cache type

SQL> select cache#, parameter from v$rowcache;

CACHE# PARAMETER---------- --------------------------------

1 dc_free_extents4 dc_used_extents2 dc_segments0 dc_tablespaces5 dc_tablespace_quotas6 dc_files7 dc_users3 dc_rollback_segments8 dc_objects17 dc_global_oids12 dc_constraints


34/41


Additional Support

AWR Tables on disk for 7 days by default

DBA_HIST_ACTIVE_SESS_HISTORY

1 in 10 ASH samples

DBA_HIST_SEG_STAT

Good for ITL and buffer busy wait

DBA_HIST_SYSTEM_EVENT

Important for getting avg wait times

DBA_HIST_SQLSTAT sql execution deltas

DBA_HIST_SYSMETRIC_SUMMARY Statistics avg, max, min

Metric Tables in memory deltas

V$EVENTMETRIC


35/41


All Events over 7 days

select count(*), event fromselect count(*), event from

( select event from DBA_HIST_ACTIVE_SESS_HISTORY( select event from DBA_HIST_ACTIVE_SESS_HISTORY

wherewhere sample_timesample_time < ( select min(< ( select min(sample_timesample_time) from) from

v$active_session_historyv$active_session_history))union allunion all

select event fromselect event from v$active_session_historyv$active_session_history

))

group by eventgroup by eventorder by eventorder by event

//


36/41


Average Wait Times Historic

select

btime,

(time_ms_end-time_ms_beg)/nullif(count_end-count_beg,0) avg_ms

from (

select

to_char(s.BEGIN_INTERVAL_TIME,'DD-MON-YY HH24:MI') btime,

total_waits count_end,

time_waited_micro/1000 time_ms_end,

LagLag ((e.time_waited_micro/1000)OVER( PARTITION BY e.event_name ORDER BY s.snap_id) time_ms_beg,

LagLag ((e.total_waits)

OVER( PARTITION BY e.event_name ORDER BY s.snap_id) count_beg

from

DBA_HIST_SYSTEM_EVENT e,

DBA_HIST_SNAPSHOT s

where

s.snap_id=e.snap_idand e.event_name= '&1'

order by begin_interval_time

)

order by btime;

BTIME AVG_MSBTIME AVG_MS

---------------------------------------- ------------------------0808--JANJAN--08 01:00 1.01708 01:00 1.017

0808--JANJAN--08 02:00 .72008 02:00 .720

0808--JANJAN--08 03:00 .62108 03:00 .621

0808--JANJAN--08 04:00 1.74708 04:00 1.747

0808--JANJAN--08 05:00 1.04608 05:00 1.046

0808--JANJAN--08 06:00 1.44408 06:00 1.444


37/41


Avg Wait times now

selectselect

en.name,en.name,

(time_waited)/nullif(wait_count,0) avg_ms,(time_waited)/nullif(wait_count,0) avg_ms,

wait_countwait_count

fromfrom

v$eventmetric e,v$eventmetric e,

v$event_name env$event_name en

wherewhere

e.event# = en.event#e.event# = en.event#

and en.name like '%&1%;and en.name like '%&1%;

NAME AVG_MS WAIT_COUNTNAME AVG_MS WAIT_COUNT-------------------------------------------------- -------------------- --------------------db file sequential read .658863707 6420db file sequential read .658863707 6420

db file scattered read .549427419 186db file scattered read .549427419 186

db file parallel write .089073438 64db file parallel write .089073438 64


38/41

Object Translation

Current fields in v$active_session_history CURRENT_OBJ#

CURRENT_FILE#

CURRENT_BLOCK#

Called ROW_WAIT_% in v$session

Only apply to Buffer Busy Waits

IO Waits

Enqueue TX

Ignore these fields for other wait events


39/41


Wait interface Weaknesses

Logons EM 10g shows these on perf page Time model helps

V$SYS_TIME_MODEL

connection management call elapsed time

Ive had problems

Paging/Memory issues CPU starvation

Null Events

Bugs read external table reports CPU http://blog.tanelpoder.com/


40/41


Dependable Tuning Strategy

Run Statspack/AWR reportTop 5 Timed Events

~50 lines down from top

Need Available CPU Elapsed Time

CPU_

COUNT

OEM 10gPerformance Page does everything !

OEM doesnt solve the problem

Query v$active_session_history directly


41/41


Summary

Waits make Tuning Easy Check Machine Health

Tune Waits

Tune CPU

Tune SQL

Change Application Architecture

Use OEM10g

Statspack/AWR,

S/ASH Ignore Background, Idle, Resmgr, PQO

Use ASH if OEM fails

See http://perfvision.com for more info

Documents

05 WAITS Intro